{"id":201,"date":"2026-10-18T21:58:34","date_gmt":"2026-10-18T13:58:34","guid":{"rendered":"https:\/\/99website.top\/blog\/?p=201"},"modified":"2026-10-25T22:32:21","modified_gmt":"2026-10-25T14:32:21","slug":"attention-is-all-you-need-5%ef%bc%9a%e7%bb%88%e4%ba%8e%e8%a7%81%e5%88%b0attention%ef%bc%81","status":"publish","type":"post","link":"https:\/\/99website.top\/blog\/?p=201","title":{"rendered":"Attention Is All You Need &#8211; #5\uff1a\u7ec8\u4e8e\u89c1\u5230Attention\uff01\uff08\u4e0a\uff09\u4ece QKV \u5230 Causal Mask"},"content":{"rendered":"\n<h1>Attention Is All You Need &#8211; #5\uff1a\u7ec8\u4e8e\u89c1\u5230Attention\uff01<\/h1>\n<blockquote>\n<p>\u4e0a\u7bc7 \u2014\u2014\u4ece QKV \u5230 Causal Mask<\/p>\n<\/blockquote>\n<p>\u547c\u3002\u3002\u3002\u300299MLP\u7684\u7814\u7a76\u6682\u65f6\u544a\u4e00\u6bb5\u843d\u5566~<\/p>\n<p>\u8fd9\u7bc7\u535a\u5ba2\u54b1\u4eec\u7ec8\u4e8e\u5f00\u59cb\u7814\u7a76Transformer\u548cAttention\u5566\uff01<\/p>\n<p><strong>\u6700\u7ec8\u7684\u76ee\u6807\u662f\u505a\u51fa\u4e00\u4e2a99GPT\uff01\uff01\uff01<\/strong><\/p>\n<h2>\u7b2c\u4e00\u5173\u6b63\u5f0f\u5f00\u59cb\uff01<\/h2>\n<p>\u5047\u8bbe\u6709\u4e00\u53e5\u8bdd\uff1a<\/p>\n<blockquote>\n<p>\u5c0f\u660e\u628a\u82f9\u679c\u653e\u8fdb\u4e66\u5305\uff0c\u56e0\u4e3a<strong>\u5b83<\/strong>\u592a\u91cd\u4e86\u3002<\/p>\n<\/blockquote>\n<p>GPT\u95ee\u5230\uff1a<\/p>\n<p>\u5982\u679c\u795e\u7ecf\u7f51\u7edc\u8981\u5224\u65ad\u300c\u5b83\u300d\u6307\u7684\u662f\u8c01\uff0c\u4f60\u89c9\u5f97\u5b83\u5e94\u8be5\u53ea\u5206\u6790\u300c\u5b83\u300d\u8fd9\u4e2a\u5b57\u672c\u8eab\uff0c\u8fd8\u662f\u5e94\u8be5\u53c2\u8003\u53e5\u5b50\u4e2d\u7684\u5176\u4ed6\u8bcd\uff1f<\/p>\n<p>\u4ee5\u53ca\uff0c\u4f60\u89c9\u5f97\u666e\u901a MLP \u80fd\u4e0d\u80fd\u5904\u7406\u8fd9\u79cd\u5173\u7cfb\uff1f\u4e3a\u4ec0\u4e48\uff1f<\/p>\n<p>hmmm\u6211\u89c9\u5f97\u80af\u5b9a\u8981\u53c2\u8003\u5176\u4ed6\u8bcd\uff0c\u56e0\u4e3a\u8fd9\u4e2a\u8bcd\u662f\u4ee3\u8bcd<\/p>\n<p>\u50cf99mlp\u90a3\u6837\u7684\uff0c\u9664\u975e\u8bad\u7ec3\u8fc7\u5f88\u591a\u6b21\uff0c\u5de8\u91cf\u7684\u8bad\u7ec3\u80fd\u8fbe\u5230\u6240\u9700\u8981\u7684\u6548\u679c<\/p>\n<p>GPT\u7ed9\u7684\u7b54\u6848\uff1a<\/p>\n<blockquote>\n<p>MLP \u53ef\u4ee5\u5b66\u4e60\u8bcd\u4e0e\u8bcd\u7684\u5173\u7cfb\u3002 \u5982\u679c\u628a\u6574\u4e2a\u53e5\u5b50\u7684\u8bcd\u5411\u91cf\u8f93\u5165\u4e00\u4e2a\u8db3\u591f\u5927\u7684 MLP\uff0c\u5b83\u786e\u5b9e\u6709\u53ef\u80fd\u901a\u8fc7\u8bad\u7ec3\u5b66\u4f1a\u5224\u65ad\u4ee3\u8bcd\u6307\u4ee3\u3002<\/p>\n<p>\u4f46\u666e\u901a MLP \u6ca1\u6709\u663e\u5f0f\u7684\u3001\u52a8\u6001\u7684\u8bcd\u95f4\u6ce8\u610f\u529b\u8ba1\u7b97\u673a\u5236\u3002 \u5b83\u901a\u5e38\u4f9d\u9760\u8bad\u7ec3\u597d\u7684\u56fa\u5b9a\u6743\u91cd\u5904\u7406\u8f93\u5165\uff0c\u800c Attention \u80fd\u6839\u636e\u5f53\u524d\u8f93\u5165\uff0c\u52a8\u6001\u8ba1\u7b97\u5404\u4e2a\u8bcd\u4e4b\u95f4\u7684\u5173\u8054\u7a0b\u5ea6\u3002<\/p>\n<\/blockquote>\n<h3>\u4e00\u4e2a\u975e\u5e38\u5173\u952e\u7684\u77e5\u8bc6\u70b9<\/h3>\n<p><strong>\u9884\u6d4b\u4e0b\u4e00\u4e2a\u8bcd\u662f\u4e00\u79cd\u8bad\u7ec3\u76ee\u6807\uff0c\u800c Attention \u662f\u4e00\u79cd\u7f51\u7edc\u8ba1\u7b97\u673a\u5236\u3002<\/strong><\/p>\n<h2>Attention \u5230\u5e95\u662f\u600e\u4e48\u505a\u5230\u7684\uff1f<\/h2>\n<p>Attention \u7684\u4e00\u4e2a\u6838\u5fc3\u601d\u8def\u5c31\u662f\uff1a<\/p>\n<p>\u8ba9\u4e00\u4e2a Token \u6839\u636e\u5f53\u524d\u4e0a\u4e0b\u6587\uff0c\u51b3\u5b9a\u5e94\u8be5\u4ece\u54ea\u4e9b Token \u83b7\u53d6\u591a\u5c11\u4fe1\u606f\u3002<\/p>\n<p>\u4e3a\u4e86\u5b9e\u73b0\u8fd9\u4e2a\u8fc7\u7a0b\uff0cTransformer \u5f15\u5165\u4e86\u4e09\u4e2a\u5411\u91cf\uff1a<\/p>\n<div class=\"table-responsive\"><table class=\"table\">\n<thead>\n<tr>\n<th>\u540d\u79f0<\/th>\n<th>\u5168\u79f0<\/th>\n<th>\u76f4\u89c2\u7406\u89e3<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Q<\/td>\n<td>Query<\/td>\n<td>\u6211\u6b63\u5728\u5bfb\u627e\u4ec0\u4e48\u4fe1\u606f\uff1f<\/td>\n<\/tr>\n<tr>\n<td>K<\/td>\n<td>Key<\/td>\n<td>\u6211\u6709\u4ec0\u4e48\u7279\u5f81\u53ef\u4f9b\u5339\u914d\uff1f<\/td>\n<\/tr>\n<tr>\n<td>V<\/td>\n<td>Value<\/td>\n<td>\u6211\u5b9e\u9645\u80fd\u591f\u63d0\u4f9b\u4ec0\u4e48\u4fe1\u606f\uff1f<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n<p>\u6ce8\u610f\uff0c\u8fd9\u53ea\u662f\u4fbf\u4e8e\u7406\u89e3\u7684\u7c7b\u6bd4\u3002<strong>Q\u3001K\u3001V \u5b9e\u9645\u4e0a\u90fd\u662f\u901a\u8fc7\u53ef\u5b66\u4e60\u7684\u6743\u91cd\u77e9\u9635\uff0c\u4ece Token \u7684\u8868\u793a\u4e2d\u8ba1\u7b97\u51fa\u6765\u7684\u3002<\/strong><\/p>\n<h3>\u9686\u91cd\u8bf7\u51fa\uff01Attention\u516c\u5f0f\uff01\uff01<\/h3>\n<div class=\"katex math multi-line\">\n\\boxed{\\operatorname{Attention}(Q,K,V)=\\operatorname{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V}\n<\/div>\n<p>\u54b1\u4eec\u5148\u7ed9\u4e2a\u5c0f\u60c5\u666f\uff0c\u5c31\u7528\u5f00\u5934\u7684\u90a3\u53e5\u8bdd\uff0c\u6709\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQ_{\\text{\u5b83}}=[1,2]\n\\\\\n\\\\\nK_{\\text{\u82f9\u679c}}=[1,0],\\quad K_{\\text{\u4e66\u5305}}=[0,1]\n<\/div>\n<p>\u4ece\u91cc\u5f80\u5916\u770b<\/p>\n<h4>QK\u70b9\u79ef\u2014\u2014\u8ba1\u7b97\u5339\u914d\u5206\u6570<\/h4>\n<ul>\n<li>\n<p><span class=\"katex math inline\">Q<\/span>\uff1aQuery\uff08\u67e5\u8be2\u77e9\u9635\uff09<\/p>\n<\/li>\n<li>\n<p><span class=\"katex math inline\">K<\/span>\uff1aKey\uff08\u952e\u77e9\u9635\uff09<\/p>\n<\/li>\n<li>\n<p><span class=\"katex math inline\">K^T<\/span>\uff1aKey \u7684\u8f6c\u7f6e\u77e9\u9635<\/p>\n<\/li>\n<li>\n<p><span class=\"katex math inline\">QK^T<\/span>\uff1a\u901a\u8fc7\u77e9\u9635\u4e58\u6cd5\uff0c\u8ba1\u7b97\u5404\u4e2a Query \u548c Key \u4e4b\u95f4\u7684\u70b9\u79ef\u76f8\u4f3c\u5ea6\u5206\u6570<\/p>\n<\/li>\n<\/ul>\n<div class=\"katex math multi-line\">\nQ\\cdot K_{\\text{\u82f9\u679c}}=1\\times1+2\\times0=1\n\\\\\nQ\\cdot K_{\\text{\u4e66\u5305}}=1\\times0+2\\times1=2\n<\/div>\n<p>\u6240\u4ee5\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\text{scores}=[1,2]\n<\/div>\n<p>\u8fd9\u5c31\u662f\u666e\u901a\u7684\u70b9\u4e58\uff0c\u77e9\u9635\u4e58\u6cd5<\/p>\n<h4>\u4e3a\u4ec0\u4e48\u8981\u9664\u4ee5<span class=\"katex math inline\">\\sqrt{d_k}<\/span>\uff1f<\/h4>\n<p><strong><span class=\"katex math inline\">d_k<\/span> \u8868\u793a Key \u5411\u91cf\u7684\u7ef4\u5ea6\uff0c\u4e5f\u5c31\u662f\u6bcf\u6b21 Q\u3001K \u70b9\u79ef\u65f6\u76f8\u4e58\u518d\u6c42\u548c\u7684\u9879\u6570\u3002<\/strong><\/p>\n<p>\u8fd9\u4e2a\u8ddf<strong>He\u521d\u59cb\u5316<\/strong>\u7684\u7406\u5ff5\u5f88\u50cf<\/p>\n<p>\u5047\u5982\u6709 <span class=\"katex math inline\">n<\/span> \u4e2a\u76f8\u4e92\u72ec\u7acb\u7684\u968f\u673a\u53d8\u91cf\uff0c\u6bcf\u4e2a\u65b9\u5dee\u4e3a 1\uff0c\u90a3\u4e48\u5b83\u4eec\u76f8\u52a0\u7684\u65b9\u5dee\u5c31\u662f <span class=\"katex math inline\">n<\/span>\u3002<\/p>\n<p>Attention \u7684\u70b9\u79ef\u662f\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQ\\cdot K=\\sum_{i=1}^{d_k}q_i k_i\n<\/div>\n<p>\u5728\u4e00\u4e2a\u7b80\u5316\u5047\u8bbe\u4e0b\uff0c\u8bbe Q\u3001K \u7684\u5404\u4e2a\u5206\u91cf\u76f8\u4e92\u72ec\u7acb\u3001\u5747\u503c\u4e3a 0\u3001\u65b9\u5dee\u4e3a 1\u3002<\/p>\n<p>\u90a3\u4e48\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\operatorname{Var}(q_i k_i)=1\n<\/div>\n<p>\u56e0\u6b64\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\operatorname{Var}(Q\\cdot K)=d_k\n<\/div>\n<blockquote>\n<p>\u8fd9\u91cc\u6211\u7ed9\u6211\u81ea\u5df1\u52a0\u4e2a\u6ce8\u89e3\uff0c\u6bcf\u4e2aq,k\u65b9\u5dee\u90fd\u662f1\uff0c\u6240\u4ee5<span class=\"katex math inline\">d_k<\/span>\u4e2aqk\u4e58\u79ef\u52a0\u5728\u4e00\u8d77\u65b9\u5dee\u5c31\u662f<span class=\"katex math inline\">d_k<\/span>\uff08\u65b9\u5dee\u7684\u6027\u8d28\uff09\uff0c\u6240\u4ee5<span class=\"katex math inline\">d_k<\/span>\u53ef\u4ee5\u8868\u8fbe\u7ef4\u5ea6<\/p>\n<\/blockquote>\n<p>\u7ef4\u5ea6\u8d8a\u5927\uff0c\u70b9\u79ef\u7684\u5178\u578b\u6ce2\u52a8\u5e45\u5ea6\u5c31\u8d8a\u5927\uff01<\/p>\n<p>\u6bd4\u5982 <span class=\"katex math inline\">d_k=64<\/span>\uff0c\u5b83\u7684\u6807\u51c6\u5dee\u5c31\u4f1a\u8fbe\u5230 8\u3002<\/p>\n<p>\u8fd9\u6837\u76f4\u63a5\u9001\u8fdb Softmax\uff0c\u5f88\u5bb9\u6613\u4ea7\u751f\u7279\u522b\u6781\u7aef\u7684\u5206\u5e03\uff0c\u4f7f\u68af\u5ea6\u53d8\u5f97\u5f88\u5c0f\u3002<\/p>\n<p>\u6240\u4ee5\u6211\u4eec\u628a\u5b83\u9664\u4ee5\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\sqrt{d_k}\n<\/div>\n<p>\u65b9\u5dee\u5c31\u53d8\u6210\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\operatorname{Var}\\left(\\frac1{\\sqrt{d_k}}{Q\\cdot K}\\right)\n=\\frac{1}{d_k}\\operatorname{Var}(Q\\cdot K)=1\n<\/div>\n<p><strong>\u8fd9\u8ddf\u6211\u7814\u7a76\u6743\u91cd\u521d\u59cb\u5316\u65f6\uff0c\u63a7\u5236\u65b9\u5dee\u7684\u601d\u60f3\u975e\u5e38\u76f8\u4f3c\uff01<\/strong><\/p>\n<p>\u56de\u5230\u6211\u4eec\u7684\u4f8b\u5b50\uff0c<span class=\"katex math inline\">d_k=2<\/span>\u3002<\/p>\n<div class=\"katex math multi-line\">\n\\text{scaled scores}\n=\\left[\\frac1{\\sqrt2},\\frac2{\\sqrt2}\\right]\n<\/div>\n<p>\u7ea6\u7b49\u4e8e\uff1a<\/p>\n<div class=\"katex math multi-line\">\n[0.707,\\ 1.414]\n<\/div>\n<h3>Softmax\u2014\u2014\u628a\u5206\u6570\u53d8\u6210\u6ce8\u610f\u529b\u6743\u91cd<\/h3>\n<p>softmax\u516c\u5f0f\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\operatorname{softmax}(z_i)\n=\\frac{e^{z_i}}{\\sum_j e^{z_j}}\n<\/div>\n<p>\uff08\u6211\u5f88\u597d\u5947\u8fd9\u4e2a\u516c\u5f0f\u5230\u5e95\u662f\u5982\u4f55\u4ea7\u751f\u7684hhhh\uff09<\/p>\n<p>\u5e26\u5165\u4e0a\u4e00\u6b65\u7b97\u597d\u7684<span class=\"katex math inline\">\\frac{Q \\cdot K}{\\sqrt{d_k}}<\/span><\/p>\n<div class=\"katex math multi-line\">\nw_{\\text{\u82f9\u679c}} =\\frac{e^{0.707}}{e^{0.707}+e^{1.414}} \\approx0.330\n<\/div>\n<div class=\"katex math multi-line\">\nw_{\\text{\u4e66\u5305}} =\\frac{e^{1.414}}{e^{0.707}+e^{1.414}} \\approx0.670\n<\/div>\n<p>\u6211\u53bb\uff01\uff1f\u52a0\u4e00\u5757\u7b49\u4e8e<code>1<\/code>\uff01\uff1f<\/p>\n<h4>\u4e3a\u4ec0\u4e48 Softmax \u8f93\u51fa\u7684\u6240\u6709\u6743\u91cd\u4e4b\u548c\u603b\u662f\u7b49\u4e8e 1\uff1f<\/h4>\n<p><strong>Softmax \u8f93\u51fa\u7684\u6570\u5b57\u52a0\u8d77\u6765\u7b49\u4e8e 1\uff0c\u4e0d\u662f\u5de7\u5408\uff0c\u800c\u662f\u5b83\u7684\u516c\u5f0f\u672c\u8eab\u5c31\u4fdd\u8bc1\u4e86\u8fd9\u4e00\u70b9\uff01<\/strong><\/p>\n<p>\u6211\u4eec\u76f4\u63a5\u4ece\u6570\u5b66\u4e0a\u8bc1\u660e\uff01<\/p>\n<div class=\"katex math multi-line\">\n\\sum_{i=1}^{n}\\operatorname{Softmax}(z_i)\n=\\sum_{i=1}^{n}\\frac{e^{z_i}}{\\sum_{j=1}^{n}e^{z_j}}\n<\/div>\n<p>\u56e0\u4e3a\u6bcf\u4e00\u9879\u7684\u5206\u6bcd\u76f8\u540c\uff0c\u53ef\u4ee5\u5408\u5e76\u6210\uff1a<\/p>\n<div class=\"katex math multi-line\">\n=\\frac{\\sum_{i=1}^{n}e^{z_i}}{\\sum_{j=1}^{n}e^{z_j}}\n<\/div>\n<p>\u6ce8\u610f\uff0c\u5206\u5b50\u548c\u5206\u6bcd\u5176\u5b9e\u662f\u5b8c\u5168\u76f8\u540c\u7684\u6c42\u548c\uff01<span class=\"katex math inline\">i<\/span>\u3001<span class=\"katex math inline\">j<\/span> \u53ea\u662f\u6c42\u548c\u4f7f\u7528\u7684\u4e0b\u6807\u3002<\/p>\n<p>\u6240\u4ee5\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{\\sum_{i=1}^{n}\\operatorname{Softmax}(z_i)=1}\n<\/div>\n<p>\u539f\u6765\u662f\u8fd9\u6837\u554a\uff0cSoftmax\u4f20\u5165\u7684\u662f\u4e00\u7ec4\u6570\u5b57\uff0c\u7136\u540e\u5206\u5b50\u662f<span class=\"katex math inline\">e<\/span>\u7684\u5f53\u524d\u5143\u7d20\u6b21\u5e42\uff0c\u7136\u540e\u5206\u6bcd\u662f\u5148\u8ba1\u7b97<span class=\"katex math inline\">e<\/span>\u7684\u6bcf\u4e2a\u5143\u7d20\u6b21\u5e42\uff0c\u518d\u628a\u8fd9\u4e9b\u7ed3\u679c\u76f8\u52a0<\/p>\n<h3>Value\u52a0\u6743\u6c42\u548c<\/h3>\n<p>\u628a\u82f9\u679c\u53e5\u5b50\u7684<code>score<\/code>\u5e26\u5165\u5230Softmax\u4e2d<\/p>\n<div class=\"katex math multi-line\">\nA=[0.330,0.670]\n<\/div>\n<p>\u8fd9\u5c31\u662f\u6ce8\u610f\u529b\u6743\u91cd\u3002<\/p>\n<p>\u4f46\u95ee\u9898\u6765\u4e86\uff1a\u5f97\u5230\u8fd9\u4e9b\u6743\u91cd\u4ee5\u540e\uff0cAttention \u7a76\u7adf\u8981\u8f93\u51fa\u4ec0\u4e48\uff1f<\/p>\n<p>\u7b54\u6848\uff1a\u901a\u8fc7\u8fd9\u4e9b\u6743\u91cd\uff0c\u5bf9\u5404\u4e2a Token \u7684 Value \u5411\u91cf\u8fdb\u884c\u52a0\u6743\u6c42\u548c\u3002<\/p>\n<p>\u5047\u8bbe\u73b0\u5728\u6709\uff1a<\/p>\n<div class=\"katex math multi-line\">\nV_{\\text{\u82f9\u679c}}=[2,0]\n\\\\\n\nV_{\\text{\u4e66\u5305}}=[0,4]\n<\/div>\n<p>\u77e9\u9635\u4e58\u6cd5\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\underbrace{\\begin{bmatrix}0.330&amp;0.670\\end{bmatrix}}_{\\text{\u6ce8\u610f\u529b\u6743\u91cd}}\n\\underbrace{\\begin{bmatrix}2&amp;0\\\\0&amp;4\\end{bmatrix}}_{\\text{Value \u77e9\u9635}}\n<\/div>\n<p>\u5f97\u5230\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{O=[0.660,2.680]}\n<\/div>\n<p>\u77e9\u9635<span class=\"katex math inline\">O<\/span>\u5c31\u662fAttention\u6700\u540e\u8f93\u51fa\u7684\u4e1c\u897f\u5566~<\/p>\n<p>GPT\u7684\u624b\u7b97\u9898\u76ee\uff1a<\/p>\n<p><img alt=\"image-20261011221246905\" src=\"https:\/\/99website.top\/blog\/wp-content\/themes\/Kizumi\/assets\/images\/loading.gif\" data-src=\"https:\/\/99website.top\/blog\/wp-content\/uploads\/2026\/10\/99blog-f88707be4b2228fddf6f.png\" class=\"lazy\" loading=\"lazy\"><\/p>\n<p>\u5f00\u5e72\uff01<\/p>\n<p><img alt=\"image-20261011221137282\" src=\"https:\/\/99website.top\/blog\/wp-content\/themes\/Kizumi\/assets\/images\/loading.gif\" data-src=\"https:\/\/99website.top\/blog\/wp-content\/uploads\/2026\/10\/99blog-60ab9789c69844a20695.png\" class=\"lazy\" loading=\"lazy\"><\/p>\n<p>\u503c\u5f97\u7eaa\u5ff5\uff01\uff01\u8fd9\u662f\u6211\u624b\u7b97\u7684\u7b2c\u4e00\u4e2aAttention\uff01<\/p>\n<h4>K\u7684\u8f6c\u7f6e<\/h4>\n<p>\u4e3a\u4ec0\u4e48\u53ea\u6709\u77e9\u9635<span class=\"katex math inline\">K<\/span>\u9700\u8981\u8f6c\u7f6e\u5462\uff1f<\/p>\n<p>\u5c31\u7528\u521a\u521a\u7684\u4f8b\u5b50\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQ=\\begin{bmatrix}1&amp;1\\end{bmatrix}\n\\qquad\nK=\\begin{bmatrix}2&amp;0\\\\0&amp;2\\end{bmatrix}\n<\/div>\n<p>\u5728\u6807\u51c6 Attention \u4e2d\uff0c<strong>\u6bcf\u4e00\u884c\u4ee3\u8868\u4e00\u4e2a Token \u7684\u5411\u91cf\u3002<\/strong>\n\u6211\u4eec\u5e0c\u671b Q \u5206\u522b\u548c K \u7684\u6bcf\u4e00\u884c\u505a\u70b9\u79ef\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQ\\cdot K_1,\\quad Q\\cdot K_2\n<\/div>\n<p>\u4f46\u77e9\u9635\u4e58\u6cd5\u662f<strong>\u884c\u4e58\u5217\uff01<\/strong><\/p>\n<p>\u6240\u4ee5\u8981\u628a K \u7684\u6bcf\u4e00\u884c\u53d8\u6210\u4e00\u5217\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQK^T= \\underbrace{\\begin{bmatrix}1&amp;1\\end{bmatrix}}_{1\\times2} \\underbrace{\\begin{bmatrix}2&amp;0\\\\0&amp;2\\end{bmatrix}}_{2\\times2}\n<\/div>\n<p>\u5f97\u5230\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{[2,2]}\n<\/div>\n<p>\u8fd9\u9053\u9898\u7684K\u6070\u597d\u8f6c\u7f6e\u524d\u540e\u5f62\u72b6\u76f8\u540c\uff0c\u4f46\u662f\u4e00\u822c\u60c5\u51b5\u4e0b\u4e0d\u662f\u8fd9\u6837<\/p>\n<p>\u4e3a\u4ec0\u4e48<span class=\"katex math inline\">V<\/span>\u4e0d\u8f6c\u7f6e\u5462\uff1f<\/p>\n<p>\u56e0\u4e3a Softmax \u5df2\u7ecf\u7ed9\u6211\u4eec\u7b97\u51fa\u4e86\u4e24\u4e2a\u6743\u91cd\uff1a<\/p>\n<div class=\"katex math multi-line\">\nS=[0.5,0.5]\n<\/div>\n<p>\u800c V \u7684\u6bcf\u4e00\u884c\u5bf9\u5e94\u4e00\u4e2a Token \u63d0\u4f9b\u7684\u4fe1\u606f\uff1a<\/p>\n<div class=\"katex math multi-line\">\nV=\\begin{bmatrix}\n2&amp;4\\\\\n6&amp;8\n\\end{bmatrix}\n<\/div>\n<p>\u6211\u4eec\u5e0c\u671b\uff1a<\/p>\n<div class=\"katex math multi-line\">\nO=0.5V_1+0.5V_2\n<\/div>\n<p>\u4e5f\u5c31\u662f\uff1a<\/p>\n<div class=\"katex math multi-line\">\nO=0.5[2,4]+0.5[6,8]\n<\/div>\n<p>\u77e9\u9635\u4e58\u6cd5\u521a\u597d\u53ef\u4ee5\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\underbrace{\\begin{bmatrix}0.5&amp;0.5\\end{bmatrix}}_{1\\times2}\n\\underbrace{\\begin{bmatrix}2&amp;4\\\\6&amp;8\\end{bmatrix}}_{2\\times2}\n=\\boxed{[4,6]}\n<\/div>\n<p><strong>\u4e00\u53e5\u8bdd\u8bb0\u5fc6\uff1aK \u8f6c\u7f6e\u662f\u4e3a\u4e86\u8ba9 Q \u9010\u4e2a\u5339\u914d Key\uff1bV \u4e0d\u8f6c\u7f6e\u662f\u4e3a\u4e86\u8ba9\u6743\u91cd\u9010\u884c\u6c47\u603b Value\u3002<\/strong><\/p>\n<h2>Self-Attention<\/h2>\n<p>\u73b0\u5728\u6211\u4eec\u4e0d\u518d\u53ea\u8003\u8651\u4e00\u4e2a Query\uff0c\u800c\u662f\u8ba9\u4e09\u4e2a Token \u90fd\u6709\u81ea\u5df1\u7684 Query\u3002<\/p>\n<p>\u5047\u8bbe\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQ\\in\\mathbb R^{3\\times2},\\quad K\\in\\mathbb R^{3\\times2}\n<\/div>\n<p>\u90a3\u4e48\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQK^T\\in\\mathbb R^{3\\times3}\n<\/div>\n<p>\u601d\u8003\u9898\uff1a\u8fd9\u4e2a <span class=\"katex math inline\">3\\times3<\/span> \u77e9\u9635\u4e2d\uff0c\u7b2c 2 \u884c\u3001\u7b2c 3 \u5217\u7684\u5143\u7d20\u4ee3\u8868\u4ec0\u4e48\uff1f<\/p>\n<p>\u6211\u7684\u7b54\u6848\uff1a<\/p>\n<p>\u7b2c 2 \u4e2a Token \u5bf9\u7b2c 3 \u4e2a Token \u7684\u5339\u914d\u5206\u6570<\/p>\n<p>\u7b54\u5bf9\u5566\uff01\uff01\uff01<\/p>\n<p>\u5047\u8bbe\u6211\u4eec\u6709\u4e09\u4e2a Token\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQ=\\begin{bmatrix}Q_1\\\\Q_2\\\\Q_3\\end{bmatrix},\n\\qquad\nK=\\begin{bmatrix}K_1\\\\K_2\\\\K_3\\end{bmatrix}\n<\/div>\n<p>\u90a3\u4e48\uff1a<\/p>\n<div class=\"katex math multi-line\">\nQK^T=\n\\begin{bmatrix}\nQ_1K_1^T&amp;Q_1K_2^T&amp;Q_1K_3^T\\\\\nQ_2K_1^T&amp;Q_2K_2^T&amp;\\boxed{Q_2K_3^T}\\\\\nQ_3K_1^T&amp;Q_3K_2^T&amp;Q_3K_3^T\n\\end{bmatrix}\n<\/div>\n<p>\u6240\u4ee5\uff1a<\/p>\n<ul>\n<li>\u884c\u4ee3\u8868\uff1a\u8c01\u6b63\u5728\u67e5\u8be2\uff1f\uff08Query\uff09<\/li>\n<li>\u5217\u4ee3\u8868\uff1a\u6b63\u5728\u67e5\u8be2\u8c01\uff1f\uff08Key\uff09<\/li>\n<li>\u7b2c 2 \u884c\u7b2c 3 \u5217\uff1a\u7b2c 2 \u4e2a Token \u5bf9\u7b2c 3 \u4e2a Token \u7684\u5339\u914d\u5206\u6570\u3002<\/li>\n<\/ul>\n<blockquote>\n<p>\u6ce8\u610f\uff0c\u8fd9\u91cc\u8fd8\u53ea\u662f\u539f\u59cb\u70b9\u79ef\u5206\u6570\uff0c\u4e0d\u662f Softmax \u540e\u7684\u6ce8\u610f\u529b\u6743\u91cd\u3002<\/p>\n<\/blockquote>\n<h3>Causal Mask<\/h3>\n<p>\u54b1\u4eec\u5df2\u7ecf\u77e5\u9053\uff0c\u6bcf\u4e2a Token \u90fd\u53ef\u4ee5\u7528 Self-Attention \u83b7\u53d6\u5176\u4ed6 Token \u7684\u4fe1\u606f\u3002<\/p>\n<p>\u4f46 GPT \u6709\u4e00\u4e2a\u9650\u5236\uff1a<\/p>\n<p><strong>\u9884\u6d4b\u4e0b\u4e00\u4e2a Token \u65f6\uff0c\u4e0d\u80fd\u5077\u5077\u770b\u5230\u672a\u6765\uff01<\/strong><\/p>\n<p>\u5047\u8bbe\u8f93\u5165\u662f\uff1a<\/p>\n<blockquote>\n<p>\u6211 \/ \u559c\u6b22 \/ \u6570\u5b66<\/p>\n<\/blockquote>\n<p>\u5bf9\u4e8e GPT\uff1a<\/p>\n<p><img alt=\"image-20261018190121177\" src=\"https:\/\/99website.top\/blog\/wp-content\/themes\/Kizumi\/assets\/images\/loading.gif\" data-src=\"https:\/\/99website.top\/blog\/wp-content\/uploads\/2026\/10\/99blog-f1c78ede7e3e11f99b74.png\" class=\"lazy\" loading=\"lazy\"><\/p>\n<p>\u56e0\u6b64\uff0c\u6211\u4eec\u9700\u8981\u4e00\u4e2a Mask \u77e9\u9635\uff1a<\/p>\n<div class=\"katex math multi-line\">\nM=\\begin{bmatrix}\n0&amp;-\\infty&amp;-\\infty\\\\\n0&amp;0&amp;-\\infty\\\\\n0&amp;0&amp;0\n\\end{bmatrix}\n<\/div>\n<p>\u5728\u8ba1\u7b97 Softmax \u4e4b\u524d\uff0c\u628a\u5b83\u52a0\u5230\u7f29\u653e\u540e\u7684\u5339\u914d\u5206\u6570\u4e0a\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{\nA=\\operatorname{softmax}\\left(\n\\frac{QK^T}{\\sqrt{d_k}}+M\n\\right)\n}\n<\/div>\n<p>\u4e3a\u4ec0\u4e48\u8981\u52a0\u8d1f\u65e0\u7a77\u5462\uff1f<\/p>\n<p>\u54b1\u4eec\u4e4b\u524d\u5df2\u7ecf\u63a8\u5bfc\u8fc7\u4e86\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\operatorname{softmax}(z_i)=\n\\frac{e^{z_i}}{\\sum_j e^{z_j}}\n<\/div>\n<p>\u800c\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\lim_{x\\to-\\infty}e^x=0\n<\/div>\n<p>\u6240\u4ee5\u88abMask\u7684\u4f4d\u7f6e\uff0cSoftmax\u4e4b\u540e\u7684\u6743\u91cd\u5c31\u53d8\u6210\u4e860\uff01\uff01<\/p>\n<p>\u8fd9\u6837\uff0c\u672a\u6765 Token \u7684 Value \u5c31\u4e0d\u4f1a\u53c2\u4e0e\u5f53\u524d Token \u7684\u52a0\u6743\u6c42\u548c\u3002<\/p>\n<p>\u6240\u4ee5\u5bf9\u4e8e\u54b1\u4eec\u7684GPT\u76ee\u6807\u6765\u8bf4\uff0c\u73b0\u5728\u66f4\u8fd1\u4e86\u4e00\u6b65\uff01<\/p>\n<p>\u8ba9\u54b1\u4eec\u6765\u5199\u4ee3\u7801\u5427~<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">import numpy as np\n\ndef self_attention(X, W_Q, W_K, W_V):\n    Q = X @ W_Q\n    K = X @ W_K\n    V = X @ W_V\n\n    # \u7b97\u7ef4\u5ea6\n    d_k = K.shape[1] #shape[1]\u8868\u793a\u53d6\u5217\u6570\uff0c\u5728\u8fd9\u91cc\u662fKey\u77e9\u9635\u7684\u7ef4\u5ea6\n    scores = (Q @ K.T) \/ np.sqrt(d_k) \n\n    # Causal Mask\n    n = X.shape[0] #shape[0]\u8868\u793a\u53d6\u884c\u6570\uff0c\u5728\u8fd9\u91cc\u53d6\u5f97\n    mask = np.triu(\n        np.ones((n, n), dtype=bool),\n        k=1\n    ) # \u751f\u6210\u4e0a\u4e09\u89d2\u906e\u7f69\n\n    scores = np.where(mask, -np.inf, scores) #\u8fdb\u884c\u906e\u7f69\u906e\u6321\n\n    scores = scores - np.max(\n        scores, axis=1, keepdims=True\n    ) #\u51cf\u53bb\u6700\u5927\u503c\uff0c\u8be6\u89c1\u535a\u5ba2\n\n    exp_scores = np.exp(scores)\n    A = exp_scores \/ np.sum(\n        exp_scores, axis=1, keepdims=True\n    )\n\n    O = A @ V\n\n    return O,A\n\n\n# \u51c6\u5907\u6570\u636e\nX = np.array([\n    [1.0, 0.0],\n    [0.0, 1.0],\n    [1.0, 1.0]\n])\n\n# \u6559\u5b66\u6f14\u793a\uff1a\u4f7f\u7528\u5355\u4f4d\u77e9\u9635\nW_Q = np.eye(2)\nW_K = np.eye(2)\nW_V = np.eye(2)\n\nO, A = self_attention(X, W_Q, W_K, W_V)\n\nnp.set_printoptions(precision=3, suppress=True)\n\nprint(&quot;\u6ce8\u610f\u529b\u6743\u91cd A:&quot;)\nprint(A)\n\nprint(&quot;Self-Attention \u8f93\u51fa O:&quot;)\nprint(O)\n\n<\/code><\/pre>\n<p>\u8f93\u51fa\u7ed3\u679c\uff1a<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-powershell\">\u6ce8\u610f\u529b\u6743\u91cd A:\n[[1.    0.    0.   ]\n [0.33  0.67  0.   ]\n [0.248 0.248 0.503]]\nSelf-Attention \u8f93\u51fa O:\n[[1.    0.   ]\n [0.33  0.67 ]\n [0.752 0.752]]\n<\/code><\/pre>\n<p>ok\u7b14\u8bb0\u65f6\u95f4\uff01<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">Q = X @ W_Q\nK = X @ W_K\nV = X @ W_V\n<\/code><\/pre>\n<p>\u8868\u793a\u8f93\u5165\u77e9\u9635\u548c\u6743\u91cdQKV\u77e9\u9635\u8fdb\u884c\u77e9\u9635\u4e58\u6cd5\u5f97\u5230QKV\u77e9\u9635<\/p>\n<div class=\"admonition note\">\n<p class=\"admonition-title\">Note<\/p>\n<p>\u8f93\u5165\u7684\u300c\u5217\u6570\u300d\u5fc5\u987b\u7b49\u4e8e\u6743\u91cd\u77e9\u9635\u7684\u300c\u884c\u6570\u300d\uff01<\/p>\n<p>\u4f46\u662fQuery\u7279\u5f81\u7ef4\u5ea6\u54b1\u4eec\u53ef\u4ee5\u81ea\u5df1\u8bbe\u8ba1\uff0c\u4e5f\u5c31\u662f<span class=\"katex math inline\">W_Q<\/span>\u7684\u5217\u6570<\/p>\n<\/div>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">d_k = K.shape[1]\n<\/code><\/pre>\n<p>\u8868\u793a\u53d6<code>K<\/code>\u77e9\u9635\u7684\u5217\u6570\uff0c\u4e5f\u5c31\u662f\u6bcf\u4e2aK\u7684\u7ef4\u5ea6<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\"># Causal Mask\n    n = X.shape[0] #shape[0]\u8868\u793a\u53d6\u884c\u6570\uff0c\u5728\u8fd9\u91cc\u53d6\u5f97\n    mask = np.triu(\n        np.ones((n, n), dtype=bool),\n        k=1\n    ) # \u751f\u6210\u4e0a\u4e09\u89d2\u906e\u7f69\n<\/code><\/pre>\n<p>\u8fd9\u6bb5\u662f\u751f\u6210<code>Causal Mask<\/code>\uff0c\u54b1\u4eec\u4e00\u5c42\u4e00\u5c42\u62c6<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">np.ones((n, n), dtype=bool)\n<\/code><\/pre>\n<p>\u610f\u601d\u662f\uff1a\u751f\u6210\u4e00\u4e2a<code>n*n<\/code>\u7684\u77e9\u9635\uff0c<code>dtype=bool<\/code>\u8868\u793a\u77e9\u9635\u91cc\u6bcf\u4e2a\u6570\u636e\u90fd\u662f\u5e03\u5c14\u503c\uff0c\u5373<code>True<\/code> \u6216 <code>False<\/code><\/p>\n<blockquote>\n<p>\u4e3a\u4ec0\u4e48 Causal Mask \u662f<code>n*n<\/code>\u7684\u77e9\u9635\u5462\uff1f\u56e0\u4e3a<span class=\"katex math inline\">Q<\/span>\u548c<span class=\"katex math inline\">K<\/span>\u77e9\u9635\u7684\u5927\u5c0f\u5b8c\u5168\u4e00\u6837\uff0c\u5747\u4e3a <span class=\"katex math inline\">(n,d_k)<\/span>\uff0cn\u662f<span class=\"katex math inline\">K<\/span>\u7684\u884c\u6570\uff0c<span class=\"katex math inline\">QK^T<\/span>\u6700\u7ec8\u662f<code>n*n<\/code>\u7684\u77e9\u9635\uff0cCausal Mask \u9700\u8981\u4e0e\u6ce8\u610f\u529b\u5206\u6570\u77e9\u9635\u9010\u5143\u7d20\u5bf9\u5e94\uff0c\u7528\u6765\u5c4f\u853d\u672a\u6765 Token\uff0c\u6240\u4ee5\u5176\u5f62\u72b6\u4e5f\u5fc5\u987b\u662f <span class=\"katex math inline\">(n,n)<\/span>\u3002<\/p>\n<\/blockquote>\n<p>\u5f97\u5230\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\begin{bmatrix}\nTrue&amp;True&amp;True\\\\\nTrue&amp;True&amp;True\\\\\nTrue&amp;True&amp;True\n\\end{bmatrix}\n<\/div>\n<p>\u7136\u540e\u6765\u770b<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">np.triu(..., k=1)\n<\/code><\/pre>\n<p><code>triu<\/code> \u662f triangle upper \u7684\u7f29\u5199\uff0c\u53ef\u4ee5\u7406\u89e3\u4e3a\u63d0\u53d6\u77e9\u9635\u7684\u4e0a\u4e09\u89d2\u90e8\u5206<\/p>\n<p>\u53c2\u6570 <code>k<\/code> \u51b3\u5b9a\u4ece\u54ea\u6761\u5bf9\u89d2\u7ebf\u5f00\u59cb<strong>\u4fdd\u7559<\/strong>\u5143\u7d20<\/p>\n<p>\u6ce8\u610f\u8fd9\u91cc\u662f\u4fdd\u7559\u5143\u7d20<\/p>\n<p>0\u662f\u4e3b\u5bf9\u89d2\u7ebf\uff0c\u8d1f\u6b63\u8868\u793a\u5411\u5de6\u5411\u53f3\u504f\u79fb\u5bf9\u89d2\u7ebf<\/p>\n<p>\u4f46\u4e3b\u5bf9\u89d2\u7ebf\u4e5f\u5305\u62ec\u5f53\u524d\u6240\u7814\u7a76\u7684token\uff0c\u6240\u4ee5<\/p>\n<p>\u6807\u51c6\u7684 GPT Causal Mask \u4f7f\u7528 <code>k=1<\/code>\uff1a<\/p>\n<div class=\"katex math multi-line\">\nmask=\n\\begin{bmatrix}\nFalse&amp;True&amp;True\\\\\nFalse&amp;False&amp;True\\\\\nFalse&amp;False&amp;False\n\\end{bmatrix}\n<\/div>\n<p>\u4f46\u7531\u4e8e<code>np.triu<\/code>\u51b3\u5b9a\u4e86\u54ea\u4e9b\u5730\u65b9\u88ab\u4fdd\u7559\uff0c\u6240\u4ee5\u54b1\u4eec\u540e\u9762\u9700\u8981\u4f7f\u7528\uff1a<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">scores = np.where(mask, -np.inf, scores) \n<\/code><\/pre>\n<p>\u610f\u601d\u662f\uff1a<\/p>\n<p>\u5982\u679c<code>mask<\/code>\u5bf9\u5e94\u4f4d\u7f6e\u662f<code>True<\/code>\uff0c\u5c31\u6539\u6210<code>-np.inf<\/code>\uff08\u8d1f\u65e0\u7a77\uff09\uff0c\u5426\u5219\u5c31\u8fd8\u4fdd\u6301<code>scores<\/code>\u672c\u8eab<\/p>\n<div class=\"admonition warning\">\n<p class=\"admonition-title\">Warning<\/p>\n<p>\u8fd9\u91cc\u8981\u6ce8\u610f\uff01\uff01\uff01\u8981\u5148\u906e\u4f4f\u672a\u6765token\u7684\u6ce8\u610f\u529b\u5206\u6570\uff0c\u5426\u5219\u5982\u679c\u5148\u51cf\u53bb\u6700\u5927\u503c\uff0c\u7136\u540eMask\uff0c\u90a3\u4e48\u8f83\u5c0f\u7684\u5206\u6570\u5728<span class=\"katex math inline\">e^x<\/span>\u8fd0\u7b97\u7684\u65f6\u5019\u5728\u5e38\u89c1\u6d6e\u70b9\u6570\u7cbe\u5ea6\u4e0b\u56de\u4e0b\u6ea2\u4e3a<code>0<\/code>\uff0c\u5bfc\u81f4\u524d\u9762\u7684token\u4e5f\u65e0\u6cd5\u8ba1\u7b97\u5206\u6570<\/p>\n<\/div>\n<p>\u90a3\u4e48\u4e3a\u4ec0\u4e48<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">scores = scores - np.max(\n        scores, axis=1, keepdims=True\n    ) #\u51cf\u53bb\u6700\u5927\u503c\uff0c\u8be6\u89c1\u535a\u5ba2\n<\/code><\/pre>\n<p>\u8fd9\u6bb5\u4ee3\u7801\u5206\u4e3a\u4e09\u90e8\u5206\uff1a<\/p>\n<ul>\n<li><code>np.max<\/code>\uff1a\u627e\u6700\u5927\u503c<\/li>\n<li><code>axis=1<\/code>\u6cbf\u7740\u5217\u7684\u65b9\u5411\uff0c\u5728<strong>\u6bcf\u4e00\u884c<\/strong>\u5185\u90e8\u627e\u6700\u5927\u503c\u3002<\/li>\n<li><code>keepdims=True<\/code>\uff1a\u4fdd\u7559\u4e8c\u4f4d\u5f62\u72b6\uff0c\u65b9\u4fbf\u540e\u7eed\u5e7f\u64ad\u51cf\u6cd5<\/li>\n<\/ul>\n<p>\u6700\u540e\u518d\u7528\u539f\u6765\u7684\u77e9\u9635\u51cf\u53bb\u6700\u5927\u503c\uff0c\u77e9\u9635\u4e2d\u7684\u6bcf\u4e2a\u5143\u7d20\u90fd\u51cf\u53bb\u8fd9\u4e00\u884c\u7684\u6700\u5927\u503c<\/p>\n<p>\u6bd4\u5982\u539f\u6765\uff1a<\/p>\n<div class=\"katex math multi-line\">\nscores=\n\\begin{bmatrix}\n1&amp;-\\infty&amp;-\\infty\\\\\n4&amp;5&amp;-\\infty\\\\\n7&amp;8&amp;9\n\\end{bmatrix}\n<\/div>\n<p>\u7136\u540e\u6c42\u6bcf\u4e00\u884c\u6700\u5927\u503c\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\begin{bmatrix}\n1\\\\5\\\\9\n\\end{bmatrix}\n<\/div>\n<p>\u5f97\u5230\uff1a<\/p>\n<div class=\"katex math multi-line\">\nscores&#8217;=\n\\begin{bmatrix}\n0&amp;-\\infty&amp;-\\infty\\\\\n-1&amp;0&amp;-\\infty\\\\\n-2&amp;-1&amp;0\n\\end{bmatrix}\n<\/div>\n<p>\u4e3a\u4ec0\u4e48\u8981\u8fd9\u6837\u505a\u5462\uff1f<\/p>\n<p>\u56e0\u4e3a Softmax \u5bf9\u6574\u7ec4\u5206\u7ec4\u540c\u65f6\u52a0\u51cf\u4e00\u4e2a\u5e38\u6570\uff0c\u7ed3\u679c\u4e0d\u53d8\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\frac{e^{z_i-c}}{\\sum_j e^{z_j-c}}\n=\\frac{e^{z_i}}{\\sum_j e^{z_j}}\n<\/div>\n<p>\u8bc1\u660e\uff1a<\/p>\n<div class=\"katex math multi-line\">\np_i=\\frac{e^{x_i}}{\\sum_j e^{x_j}}\n<\/div>\n<p>\u54b1\u4eec\u7ed9\u6240\u6709\u7684\u8f93\u5165\u51cf\u53bb\u540c\u4e00\u4e2a\u6570<span class=\"katex math inline\">c<\/span><\/p>\n<div class=\"katex math multi-line\">\np&#8217;_i=\\frac{e^{x_i-c}}{\\sum_j e^{x_j-c}}\n<\/div>\n<p>\u5229\u7528\uff1a<\/p>\n<div class=\"katex math multi-line\">\ne^{a-b}=\\frac{e^a}{e^b}\n<\/div>\n<p>\u6709\uff1a<\/p>\n<div class=\"katex math multi-line\">\np&#8217;_i=\n\\frac{e^{x_i}\/e^c}{\\sum_j e^{x_j}\/e^c}\n<\/div>\n<p>\u540c\u65f6\u7ea6\u6389<span class=\"katex math inline\">1\/e^c<\/span>\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{p&#8217;_i=\\frac{e^{x_i}}{\\sum_j e^{x_j}}=p_i}\n<\/div>\n<p>\u8bc1\u6bd5<\/p>\n<p>\u51cf\u53bb\u6700\u5927\u503c\u80fd\u591f\u9632\u6b62 <span class=\"katex math inline\">e^x<\/span> \u8fc7\u5927\uff0c\u5bfc\u81f4\u8ba1\u7b97\u673a\u6d6e\u70b9\u6570\u6ea2\u51fa\u3002<\/p>\n<p>\u8fd9\u53eb<strong>\u6570\u503c\u7a33\u5b9a\u6027\u4f18\u5316\uff01<\/strong><\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">exp_scores = np.exp(scores)\n<\/code><\/pre>\n<p><code>np.exp(x)<\/code>\u5c31\u662f\u8ba1\u7b97<\/p>\n<div class=\"katex math multi-line\">\ne^x\n<\/div>\n<p>\u5b83\u4f1a\u5bf9\u77e9\u9635\u4e2d\u6bcf\u4e2a\u5143\u7d20\u5206\u522b\u8fd0\u7b97<\/p>\n<p>\u6240\u4ee5\uff1a<\/p>\n<div class=\"katex math multi-line\">\nexp\\_scores=\n\\begin{bmatrix}\n1&amp;0&amp;0\\\\\ne^{-1}&amp;1&amp;0\\\\\ne^{-2}&amp;e^{-1}&amp;1\n\\end{bmatrix}\n<\/div>\n<p>\u8981\u6ce8\u610f<\/p>\n<div class=\"katex math multi-line\">\ne^{-\\infty}=0\n<\/div>\n<p>\u7b49\u4e8e<code>0<\/code>\u54e6<\/p>\n<p>\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48\u88ab Causal Mask \u906e\u6321\u7684 Token \u6700\u7ec8\u6743\u91cd\u4e3a 0\u3002<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">A = exp_scores \/ np.sum(\n    exp_scores, axis=1, keepdims=True\n)\n<\/code><\/pre>\n<p>\u5b9e\u73b0 Softmax \u516c\u5f0f\uff01<\/p>\n<pre class=\"prettyprint linenums\"><code class=\"language-python\">np.sum(exp_scores, axis=1, keepdims=True)\n<\/code><\/pre>\n<ul>\n<li><code>axis=1<\/code>\uff1a\u6309\u884c\u6c42\u548c<\/li>\n<li><code>keepdims=True<\/code>\uff1a\u4fdd\u7559\u4e8c\u7ef4\u7ed3\u6784\uff0c\u7ed3\u679c\u4ecd\u7136\u662f\u77e9\u9635<\/li>\n<\/ul>\n<p>\u6700\u540e\u54b1\u4eec\u5c31\u5f97\u5230\u4e86\u54b1\u4eec\u7684<code>A<\/code>\u77e9\u9635\u5566~\u548c<code>V<\/code>\u505a\u77e9\u9635\u4e58\u6cd5\u5c31\u5f97\u5230\u4e86\u6700\u540e\u7684<code>O<\/code>\u77e9\u9635\uff01\uff01<\/p>\n<h2>Self-Attention\u7684\u53cd\u5411\u4f20\u64ad<\/h2>\n<p>\u5567\uff0c\u5199\u5230\u8fd9\u91cc\u6211\u7a81\u7136\u53d1\u73b0\u4e86\u4e00\u4e2a\u95ee\u9898\uff0c\u53cd\u5411\u4f20\u64ad\u80af\u5b9a\u9700\u8981\u8ba1\u7b97Loss\uff0cLoss\u53c8\u8981\u901a\u8fc7\u6b63\u786e\u7b54\u6848\u6765\u8ba1\u7b97\uff0c\u53ef\u662f\u6211\u4e5f\u4e0d\u77e5\u9053\u6b63\u786e\u7b54\u6848\u662f\u5565\u554a\uff01<\/p>\n<p>\u6211\u8fd8\u4e0d\u77e5\u9053\u6587\u5b57\u600e\u4e48\u53d8\u6210\u5411\u91cf\u5462\uff01<\/p>\n<p>\u6240\u4ee5\u6211\u6253\u7b97\u8865\u4e00\u4e2a\u7ae0\u8282<\/p>\n<p>#4.5\u300a\u4ece\u6587\u5b57\u5230\u5411\u91cf\uff1aTokenizer \u4e0e Token Embedding\u300b<\/p>\n<p>\u7b49\u6211\u56de\u6765\uff01<\/p>\n<p>2026.10.25<\/p>\n<p>hhhh\u56de\u6765\u5566~<\/p>\n<p><a href=\"https:\/\/99website.top\/blog\/?p=204\">Attention is all you need &#8211; #4.5 Tokenizer \u4e0e Embedding &#8211; 99\u7684\u5c0f\u7ad9 &#8211; 99\u7684\u5c0f\u7ad9<\/a><\/p>\n<p>\u5feb\u53bb\u770b\u770b\u5427~<\/p>\n<p>hhh\u56de\u5230\u6b63\u9898\uff0c\u65e2\u7136\u662f\u53cd\u5411\u4f20\u64ad\uff0c\u90a3\u54b1\u4eec\u5c31\u8981\u4ece\u524d\u5411\u4f20\u64ad\u53cd\u7740\u4e00\u6b65\u4e00\u6b65\u8d70\u56de\u53bb<\/p>\n<p>\u524d\u5411\u4f20\u64ad\u662f\u6839\u636eQ,K,V\u548cX\u8ba1\u7b97\u7684\uff0cX\u53c8\u6765\u81eaEmbedding\u77e9\u9635\uff0c\u6240\u4ee5\u54b1\u4eec\u5728\u53cd\u5411\u4f20\u64ad\u9700\u8981\u63a8\u5bfc<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{\\frac{\\partial L}{\\partial E},\n\\quad\\frac{\\partial L}{\\partial W_Q},\n\\quad\\frac{\\partial L}{\\partial W_K},\n\\quad\\frac{\\partial L}{\\partial W_V}}\n<\/div>\n<p>\u4e5f\u5c31\u662f\u6c42\u51fa Embedding\u3001Q\u3001K\u3001V \u6295\u5f71\u77e9\u9635\u7684\u53c2\u6570\u68af\u5ea6\u3002<\/p>\n<p>\u8fd9\u6b21\u54b1\u4eec\u5148\u5047\u8bbe\uff1a<\/p>\n<ul>\n<li>\n<p>\u8bcd\u8868\u5927\u5c0f\uff1a6<\/p>\n<\/li>\n<li>\n<p>\u8f93\u5165 Token \u6570\uff1a<span class=\"katex math inline\">n=3<\/span><\/p>\n<\/li>\n<li>\n<p>Embedding \u7ef4\u5ea6\uff1a<span class=\"katex math inline\">d_{\\text{model}}=4<\/span><\/p>\n<\/li>\n<li>\n<p>Q\u3001K \u5411\u91cf\u7ef4\u5ea6\uff1a<span class=\"katex math inline\">d_k=2<\/span><\/p>\n<\/li>\n<li>\n<p>V \u5411\u91cf\u7ef4\u5ea6\uff1a<span class=\"katex math inline\">d_v=2<\/span><\/p>\n<\/li>\n<\/ul>\n<div class=\"table-responsive\"><table class=\"table\">\n<thead>\n<tr>\n<th>\u77e9\u9635<\/th>\n<th>\u7ef4\u5ea6<\/th>\n<th>\u542b\u4e49<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><span class=\"katex math inline\">E<\/span><\/td>\n<td><span class=\"katex math inline\">6\\times4<\/span><\/td>\n<td>Embedding \u53c2\u6570<\/td>\n<\/tr>\n<tr>\n<td><span class=\"katex math inline\">X<\/span><\/td>\n<td><span class=\"katex math inline\">3\\times4<\/span><\/td>\n<td>Token \u5411\u91cf<\/td>\n<\/tr>\n<tr>\n<td><span class=\"katex math inline\">W_Q,W_K<\/span><\/td>\n<td><span class=\"katex math inline\">4\\times2<\/span><\/td>\n<td>Q\u3001K \u6295\u5f71\u6743\u91cd<\/td>\n<\/tr>\n<tr>\n<td><span class=\"katex math inline\">W_V<\/span><\/td>\n<td><span class=\"katex math inline\">4\\times2<\/span><\/td>\n<td>V \u6295\u5f71\u6743\u91cd<\/td>\n<\/tr>\n<tr>\n<td><span class=\"katex math inline\">Q,K,V<\/span><\/td>\n<td><span class=\"katex math inline\">3\\times2<\/span><\/td>\n<td>\u6ce8\u610f\u529b\u8ba1\u7b97\u8f93\u5165<\/td>\n<\/tr>\n<tr>\n<td><span class=\"katex math inline\">S,A<\/span><\/td>\n<td><span class=\"katex math inline\">3\\times3<\/span><\/td>\n<td>\u6ce8\u610f\u529b\u5206\u6570\u4e0e\u6743\u91cd<\/td>\n<\/tr>\n<tr>\n<td><span class=\"katex math inline\">O<\/span><\/td>\n<td><span class=\"katex math inline\">3\\times2<\/span><\/td>\n<td>Attention \u8f93\u51fa<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n<h3>\u4ece\u8f93\u51faO\u53cd\u5411\u4f20\u64ad<\/h3>\n<p>\u6211\u4eec\u77e5\u9053\uff1a<\/p>\n<div class=\"katex math multi-line\">\nO=AV\n<\/div>\n<p>\u5047\u8bbe\u540e\u9762\u7684\u7f51\u7edc\u5df2\u7ecf\u4f20\u56de\u68af\u5ea6\uff1a<\/p>\n<div class=\"katex math multi-line\">\nG_O=\\frac{\\partial L}{\\partial O}\n<\/div>\n<p>\u90a3\u600e\u4e48\u7b97 <span class=\"katex math inline\">A<\/span> \u548c <span class=\"katex math inline\">V<\/span> \u7684\u68af\u5ea6\u5462\uff1f<\/p>\n<p>\u77e9\u9635\u53cd\u5411\u4f20\u64ad\u4f9d\u7136\u9075\u5faa\u94fe\u5f0f\u6cd5\u5219\u3002\u4f46\u4e00\u4e2a\u5143\u7d20\u53ef\u80fd\u901a\u8fc7\u591a\u6761\u8def\u5f84\u5f71\u54cd Loss\uff0c\u6240\u4ee5\u9700\u8981\u628a\u5404\u6761\u8def\u5f84\u8d21\u732e\u7684\u68af\u5ea6\u76f8\u52a0\u3002\u77e9\u9635\u8fd0\u7b97\u53ef\u4ee5\u9ad8\u6548\u5730\u5b8c\u6210\u8fd9\u4e9b\u8ba1\u7b97\u3002<\/p>\n<p>\u770b\u4e00\u4e2a\u7b80\u5316\u7248 Attention\uff1a<\/p>\n<div class=\"katex math multi-line\">\nA=[a_1,a_2]=[0.3,0.7]\n<\/div>\n<div class=\"katex math multi-line\">\nV=\\begin{bmatrix}2&amp;4\\\\6&amp;8\\end{bmatrix}\n<\/div>\n<p>\u524d\u5411\u4f20\u64ad\uff1a<\/p>\n<div class=\"katex math multi-line\">\nO=AV =\\begin{bmatrix}4.8&amp;6.8\\end{bmatrix}\n<\/div>\n<p>\u73b0\u5728\u6211\u4eec\u5047\u8bbe\u4e00\u4e2a\u6559\u5b66\u7528\u7684 Loss\uff1a<\/p>\n<div class=\"katex math multi-line\">\nL=O_1+2O_2\n<\/div>\n<p>\u90a3\u4e48\u4e0a\u6e38\u4f20\u56de\u6765\u7684\u68af\u5ea6\u5c31\u662f\uff1a<\/p>\n<div class=\"katex math multi-line\">\nG_O=\\frac{\\partial L}{\\partial O}\n=\\begin{bmatrix}1&amp;2\\end{bmatrix}\n<\/div>\n<p>\u8fd9\u53e5\u8bdd\u5f88\u91cd\u8981\uff1a<strong>\u5b83\u8868\u793a <span class=\"katex math inline\">O_1<\/span> \u6539\u53d8\u4e00\u70b9\uff0c\u5bf9 Loss \u7684\u5f71\u54cd\u7cfb\u6570\u662f 1\uff1b<span class=\"katex math inline\">O_2<\/span> \u7684\u5f71\u54cd\u7cfb\u6570\u662f 2\u3002<\/strong><\/p>\n<h5>\u6c42 A \u7684\u68af\u5ea6<\/h5>\n<p>\u54b1\u4eec\u628a\u77e9\u9635\u4e58\u6cd5\u5c55\u5f00\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\begin{cases}\nO_1=2a_1+6a_2\\\\\nO_2=4a_1+8a_2\n\\end{cases}\n<\/div>\n<p>\u6ce8\u610f\uff01<span class=\"katex math inline\">a_1<\/span> \u540c\u65f6\u5f71\u54cd\u4e86 <span class=\"katex math inline\">O_1<\/span> \u548c <span class=\"katex math inline\">O_2<\/span>\u3002<\/p>\n<p>\u6240\u4ee5\u5b83\u5bf9 Loss \u7684\u5f71\u54cd\u9700\u8981\u628a\u4e24\u6761\u8def\u5f84<strong>\u52a0\u8d77\u6765<\/strong>\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\frac{\\partial L}{\\partial a_1}\n=\n\\frac{\\partial L}{\\partial O_1}\n\\frac{\\partial O_1}{\\partial a_1}\n+\n\\frac{\\partial L}{\\partial O_2}\n\\frac{\\partial O_2}{\\partial a_1}\n<\/div>\n<p>\u4ee3\u5165\uff1a<\/p>\n<div class=\"katex math multi-line\">\n=1\\times2+2\\times4\n=\\boxed{10}\n<\/div>\n<p>\u540c\u7406\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\frac{\\partial L}{\\partial a_2}\n=1\\times6+2\\times8\n=\\boxed{22}\n<\/div>\n<blockquote>\n<p>\u5728\u8fd9\u4e2a\u4f8b\u5b50\u4e2d\uff0c<span class=\"katex math inline\">O_1<\/span>\u3001<span class=\"katex math inline\">O_2<\/span> \u5bf9 <span class=\"katex math inline\">a_1<\/span>\u3001<span class=\"katex math inline\">a_2<\/span> \u90fd\u662f\u7ebf\u6027\u51fd\u6570\u3002\u6c42\u67d0\u4e2a\u53d8\u91cf\u7684\u504f\u5bfc\u65f6\uff0c\u53ef\u4ee5\u4fdd\u6301\u5176\u4ed6\u53d8\u91cf\u4e0d\u53d8\uff0c\u76f4\u63a5\u53d6\u5bf9\u5e94\u7684\u7cfb\u6570\u3002<\/p>\n<\/blockquote>\n<p>\u56e0\u6b64\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{G_A=\\begin{bmatrix}10&amp;22\\end{bmatrix}}\n<\/div>\n<p>\u8f6c\u6362\u6210\u516c\u5f0f\u5462\uff1f<\/p>\n<p>\u5df2\u77e5\uff1a<\/p>\n<div class=\"katex math multi-line\">\nG_O=\\begin{bmatrix}1&amp;2\\end{bmatrix}\n\n\\\\\n\\\\\n\nV=\\begin{bmatrix}2&amp;4\\\\6&amp;8\\end{bmatrix}\n<\/div>\n<p>\u6211\u4eec\u5e0c\u671b\uff1a<\/p>\n<ul>\n<li>\u7b2c\u4e00\u4e2a\u68af\u5ea6\u662f <span class=\"katex math inline\">1\\times2+2\\times4=10<\/span><\/li>\n<li>\u7b2c\u4e8c\u4e2a\u68af\u5ea6\u662f <span class=\"katex math inline\">1\\times6+2\\times8=22<\/span><\/li>\n<\/ul>\n<p>\u4e5f\u5c31\u662f\u8bf4\uff0c\u6211\u4eec\u9700\u8981\u8ba9 <span class=\"katex math inline\">G_O<\/span> \u5206\u522b\u548c V \u7684<strong>\u6bcf\u4e00\u884c<\/strong>\u8fdb\u884c\u70b9\u79ef\uff01<\/p>\n<p>\u6240\u4ee5\u8f6c\u7f6e\uff01<\/p>\n<div class=\"katex math multi-line\">\nV^T=\\begin{bmatrix}2&amp;6\\\\4&amp;8\\end{bmatrix}\n<\/div>\n<p>\u7136\u540e\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\begin{aligned}\nG_A&amp;=G_OV^T\\\\\n&amp;=\\begin{bmatrix}1&amp;2\\end{bmatrix}\n\\begin{bmatrix}2&amp;6\\\\4&amp;8\\end{bmatrix}\\\\\n&amp;=\\boxed{\\begin{bmatrix}10&amp;22\\end{bmatrix}}\n\\end{aligned}\n<\/div>\n<p><strong>\u6211\u53bb\uff01\u8fd9\u4e0d\u5c31\u662f\u521a\u521a\u7528\u94fe\u5f0f\u6cd5\u5219\u7b97\u51fa\u6765\u7684\u7ed3\u679c\u561b\uff01<\/strong><\/p>\n<p>\u4e8e\u662f\u6211\u4eec\u5f97\u5230\u4e86\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{\\frac{\\partial L}{\\partial A}\n=\\frac{\\partial L}{\\partial O}V^T}\n<\/div>\n<p>\u8fd9\u5c31\u662f<span class=\"katex math inline\">A<\/span>\u7684\u68af\u5ea6<\/p>\n<h5><span class=\"katex math inline\">V<\/span>\u7684\u68af\u5ea6<\/h5>\n<p>\u6bd4\u5982 <span class=\"katex math inline\">V_{1,1}=2<\/span>\uff0c\u5b83\u53c2\u4e0e\u7684\u662f\uff1a<\/p>\n<div class=\"katex math multi-line\">\nO_1=a_1V_{1,1}+a_2V_{2,1}\n<\/div>\n<p>\u6240\u4ee5\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\frac{\\partial L}{\\partial V_{1,1}}\n=\n\\frac{\\partial L}{\\partial O_1}a_1\n=1\\times0.3=0.3\n<\/div>\n<p>\u5176\u4ed6\u4f4d\u7f6e\u540c\u7406\uff0c\u6700\u7ec8\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\frac{\\partial L}{\\partial V_{1,2}}\n=\\frac{\\partial L}{\\partial O_2}\\cdot a_1\n=2\\times0.3=0.6\n<\/div>\n<p>\u89c2\u5bdf\u4e00\u4e0b\uff0c\u5b83\u6070\u597d\u7b49\u4e8e\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{G_V=A^TG_O}\n<\/div>\n<p>\u56e0\u6b64\uff0c\u5bf9\u4e8e <span class=\"katex math inline\">O=AV<\/span>\uff0c\u54b1\u4eec\u7ec8\u4e8e\u5f97\u5230\u4e86\u4e24\u6761\u53cd\u5411\u4f20\u64ad\u516c\u5f0f\uff1a<\/p>\n<div class=\"katex math multi-line\">\n\\boxed{G_A=G_OV^T}\n\n\\\\\n\\\\\n\n\\boxed{G_V=A^TG_O}\n<\/div>\n<p>\u4e0b\u6b21\uff0c\u54b1\u4eec\u7ee7\u7eed\u7814\u7a76\u4ece<span class=\"katex math inline\">G_V<\/span>\u63a8\u5bfc\u5230<span class=\"katex math inline\">G_{W_V}<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Attention Is All You Need &#8211; #5\uff1a\u7ec8\u4e8e\u89c1\u5230Attention\uff01 \u4e0a\u7bc7 \u2014\u2014\u4ece QKV \u5230 Causal Mask \u547c\u3002\u3002\u3002\u300299MLP\u7684\u7814\u7a76\u6682\u65f6\u544a\u4e00\u6bb5\u843d\u5566~ \u8fd9\u7bc7\u535a\u5ba2\u54b1\u4eec\u7ec8\u4e8e\u5f00\u59cb\u7814\u7a76Transformer\u548cAttention\u5566\uff01 \u6700\u7ec8\u7684\u76ee\u6807\u662f\u505a\u51fa\u4e00\u4e2a99GPT\uff01\uff01\uff01 \u7b2c\u4e00\u5173\u6b63\u5f0f\u5f00\u59cb\uff01 \u5047\u8bbe\u6709\u4e00\u53e5\u8bdd\uff1a \u5c0f\u660e\u628a\u82f9\u679c\u653e\u8fdb\u4e66\u5305\uff0c\u56e0\u4e3a\u5b83\u592a\u91cd\u4e86\u3002 GPT\u95ee\u5230\uff1a \u5982\u679c\u795e\u7ecf [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25,23,26,24],"tags":[],"class_list":["post-201","post","type-post","status-publish","format-standard","hentry","category-ai","category-transformer","category-26","category-24"],"_links":{"self":[{"href":"https:\/\/99website.top\/blog\/index.php?rest_route=\/wp\/v2\/posts\/201","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/99website.top\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/99website.top\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/99website.top\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/99website.top\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=201"}],"version-history":[{"count":3,"href":"https:\/\/99website.top\/blog\/index.php?rest_route=\/wp\/v2\/posts\/201\/revisions"}],"predecessor-version":[{"id":205,"href":"https:\/\/99website.top\/blog\/index.php?rest_route=\/wp\/v2\/posts\/201\/revisions\/205"}],"wp:attachment":[{"href":"https:\/\/99website.top\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=201"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/99website.top\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=201"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/99website.top\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=201"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}