Ê׸ö²âÊÔʱ¹²½ø»¯ºÏ³É¿ò¼ÜTTCS£ºÔÚ¡¸×óÓÒ»¥²«¡¹ÖÐÍ»ÆÆÍÆÀíÆ¿¾±
2026-03-01 08:24:06

ÔÚ DeepSeek-R1 ºÍ OpenAI o1 ÒýÁìµÄ¡¸ºóѵÁ·£¨Post-Training£©¡¹Ó롸²âÊÔʱÀ©Õ¹¡¹£¨Test-Time Scaling£©¡¹Ê±´ú£¬£¬£¬£¬ £¬ÔõÑùʹÓòâÊÔʱµÄËãÁ¦¾ÙÐÐÓÐÓÃѵÁ·³ÉΪ½¹µã¡£¡£¡£¡£¡£

È»¶ø£¬£¬£¬£¬ £¬ÃæÁÙ¼«ÄѵIJâÊÔÌ⣬£¬£¬£¬ £¬ÏÖÓеIJâÊÔʱѵÁ·£¨Test-Time Training, TTT£©ÍùÍùÒòα±êÇ©ÔëÉù´ó¶øÏÝÈ롸Ϲ²Â¡¹µÄÄæ¾³¡£¡£¡£¡£¡£

ÏÃÃÅ´óѧ DeepLIT ¿ÎÌâ×éÌá³öÁËÒ»ÖÖȫеIJâÊÔʱ¿Î³ÌºÏ³É¿ò¼Ü ¡ª¡ªTTCS (Test-Time Curriculum Synthesis)¡£¡£¡£¡£¡£¸Ã¿ò¼Ü²»ÒÀÀµÈκÎÍⲿÈ˹¤±ê×¢£¬£¬£¬£¬ £¬Í¨¹ýÌìÉúÆ÷£¨Synthesizer£©ÓëÇó½âÆ÷£¨Solver£©µÄ¹²½ø»¯²©ÞÄ£¬£¬£¬£¬ £¬×Ô¶¯ºÏ³É´¦ÓÚÄ£×Ó¡¸ÄÜÁ¦½çÏß¡¹µÄ¿Î³ÌÊý¾Ý£¬£¬£¬£¬ £¬½â¾öÁ˲âÊÔÑù±¾¹ýÄѵ¼ÖµÄѵÁ·Ì®ËúÎÊÌâ¡£¡£¡£¡£¡£

ʵÑéÏÔʾ£¬£¬£¬£¬ £¬ÔÚ Qwen2.5-Math-1.5B ÉÏ£¬£¬£¬£¬ £¬TTCS ½«ÊýÑ§ÍÆÀíÆ½¾ù·Ö´Ó 17.30 ±©ÕÇÖÁ 41.49£¬£¬£¬£¬ £¬ÔÚ¼«¾ßÌôÕ½µÄ AIME ¾ºÈüÌâÉϸüÊÇÏÔÖøÓâÔ½ÁË TTRL µÈÇ¿»ùÏß¡£¡£¡£¡£¡£

ÂÛÎÄÎÊÌ⣺TTCS: Test-Time Curriculum Synthesis for Self-EvolvingÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/abs/2601.22628ÏîÄ¿´úÂ룺https://github.com/XMUDeepLIT/TTCSHuggingFace Ö÷Ò³£ºhttps://huggingface.co/papers/2601.22628

01. ½¹µãÄîÍ·£ºµ±¡¸Ì⺣սÊõ¡¹Óöµ½¡¸³¬¸ÙÄÑÌ⡹

Ëæ×Å´óÓïÑÔÄ£×Ó£¨LLM£©µÄÉú³¤£¬£¬£¬£¬ £¬Òµ½ç¹²Ê¶ÒÑ´Ó´¿´âµÄ¡¸Ô¤ÑµÁ·À©´ó²ÎÊý¡¹×ªÏòÍÚ¾ò²âÊÔʱÀ©Õ¹£¨Test-Time Scaling£©µÄDZÁ¦¡£¡£¡£¡£¡£

DeepSeek-R1 µÈÄ£×ÓµÄÀÖ³É֤ʵÎúÇ¿»¯Ñ§Ï°£¨RL£©ÔÚÍÆÀíʹÃüÉϵÄÖØ´óÍþÁ¦¡£¡£¡£¡£¡£È»¶ø£¬£¬£¬£¬ £¬ÏÖÔÚµÄ RL ·¶Ê½£¨Èç RLVR£©Í¨³£ÒÀÀµ´ó×Ú¸ßÖÊÁ¿µÄ Ground Truth ±êÇ©£¬£¬£¬£¬ £¬ÕâÏÞÖÆÁËÆäÀ©Õ¹ÐÔ¡£¡£¡£¡£¡£

ΪÁËÕõÍѶԱêÇ©µÄÒÀÀµ£¬£¬£¬£¬ £¬²âÊÔʱѵÁ·£¨Test-Time Training, TTT£©Ó¦Ô˶øÉú¡£¡£¡£¡£¡£Æä½¹µãÍ·ÄÔÊÇ£ºÔÚ²âÊԽ׶Σ¬£¬£¬£¬ £¬Ê¹ÓÃÄ£×Ó¶Ô²âÊÔÌâÌìÉúµÄα±êÇ©£¨Pseudo-labels£©¾ÙÐм´Ê±¸üС£¡£¡£¡£¡£

µ«ÔÚÃæÁÙ¸ßÄѶÈÍÆÀíʹÃü£¨Èç AIME Êýѧ¾ºÈü£©Ê±£¬£¬£¬£¬ £¬TTT ÔâÓöÁËÖÂÃüµÄ¡¸ÄÜÁ¦´íÅ䡹ã£ÂÛ£º

α±êÇ©²»¿É¿¿£¨Noisy Rewards£©£ºÃæÁÙ AIME ÕâÖÖÄÑÌ⣬£¬£¬£¬ £¬ÓÉÓÚÄ£×Ó×Ô¼ºÄÜÁ¦È±·¦£¬£¬£¬£¬ £¬×ÝÈ»¾ÙÐжà´Î²ÉÑù£¨Majority Voting£©£¬£¬£¬£¬ £¬´ó²¿·ÖÃÕµ×Ò²ÊÇ´íµÄ¡£¡£¡£¡£¡ £»£»£»£»£» £»£»£»ùÓÚ¹ýʧµÄ¹²Ê¶¾ÙÐÐѵÁ·£¬£¬£¬£¬ £¬Ö»»áÈÃÄ£×Ó¡¸Ô½Ñ§Ô½Æ«¡¹¡£¡£¡£¡£¡££¨Èç Intro ÖÐ Figure 1 (a) Ëùʾ£©È±·¦ÖÐÐÄõè¾¶£¨Steep Learning Curve£©£º¾ÍÏñÈÃСѧÉúÖ±½Ó×ö΢»ý·Ö£¬£¬£¬£¬ £¬È±·¦ÖÐÐÄÄѶȵĹý¶ÉÎÊÌ⣬£¬£¬£¬ £¬Ä£×ÓÎÞ·¨¿çÔ½ÖØ´óµÄÄѶȺ蹵£¬£¬£¬£¬ £¬µ¼ÖÂÓÅ»¯Ê§°Ü¡£¡£¡£¡£¡£

ÍŶÓ˼Ë÷£ºÈôÊÇûÓÐÏÈÉú£¬£¬£¬£¬ £¬Ä£×ÓÄÜ·ñÏñÈËÀà×ÔѧÕßÒ»Ñù£¬£¬£¬£¬ £¬Í¨¹ý×Ô¼º¸ø×Ô¼º³öÌ⣬£¬£¬£¬ £¬°ÑÒ»µÀÄÑÌâ²ð½â±äΪһ×é³ÉÌݶȡ¢¿É½â¾öµÄѵÁ·Ì⣬£¬£¬£¬ £¬´Ó¶øÊµÏÖÂÝÐýÉÏÉý£¿£¿£¿£¿£¿£¿

»ùÓÚ´Ë£¬£¬£¬£¬ £¬ÍŶÓÌá³öÁËTTCS¡ª¡ª Ò»¸ö»ùÓÚ¹²½ø»¯£¨Co-evolution£©µÄ²âÊÔʱ¿Î³ÌºÏ³É¿ò¼Ü¡£¡£¡£¡£¡£

02. ÒªÁìÂÛ£ºSynthesizer Óë Solver µÄË«ÖØ¹²½ø»¯

TTCS µÄ½¹µãÔÚÓÚ¹¹½¨ÁËÒ»¸ö±Õ»·µÄÉú̬ϵͳ£¬£¬£¬£¬ £¬°üÀ¨Á½¸ö¹²Ïí³õÊ¼È¨ÖØµÄ Agent£¬£¬£¬£¬ £¬ËüÃÇͨ¹ýµü´úʽµÄGRPO (Group Relative Policy Optimization)¾ÙÐÐѵÁ·£º

½ÇÉ«·Ö¹¤

Òªº¦»úÖÆ£ºÔõÑùѰÕÒ¡¸ÄÜÁ¦½çÏß¡¹£¿£¿£¿£¿£¿£¿

ΪÁ˱ÜÃâÌìÉúµÄÎÊÌâÌ«¼òÆÓ£¨ÎÞЧѵÁ·£©»òÌ«ÄÑ£¨ÎÞЧ·´Ï죡£¡£¡£¡£©£¬£¬£¬£¬ £¬ÍŶÓÉè¼ÆÁËÒ»Ì×ϸÃܵÄÄÜÁ¦×Ô˳Ӧ½±Àø£¨Capability-Adaptive Reward£©»úÖÆ£º

±Õ»·Ð§Ó¦£º

Solver ÄÜÁ¦ÌáÉýºó£¬£¬£¬£¬ £¬Æä¡¸ÄÜÁ¦½çÏß¡¹»áÏò¸üÄѵÄÇøÓòÒÆ¶¯£¬£¬£¬£¬ £¬¸ÐÖªµ½Õâһת±äµÄ Synthesizer ΪÁË»ñµÃ¸ß½±Àø»á×Ô¶¯ÌìÉúÄѶȸü¸ßµÄÎÊÌ⣬£¬£¬£¬ £¬Solver ¼ÌÐø¹¥¿ËÐÂÄÑÌâ¡£¡£¡£¡£¡£¶þÕßÏ໥׷¸Ï£¬£¬£¬£¬ £¬ÔÚ²âÊÔʱʵÏÖÁ˶¯Ì¬µÄ Curriculum Learning¡£¡£¡£¡£¡£

03. Ó²ºËʵÑéЧ¹û£ºÍÆÀíÄÜÁ¦ÌáÉý

ÍŶÓÔÚ AMC23¡¢AIME24¡¢AIME25¡¢MATH-500¡¢Minerva µÈ¶à¸öȨÍþÊýѧ»ù×¼ÉϾÙÐÐÁËÆÕ±éÑéÖ¤¡£¡£¡£¡£¡ £»£»£»£»£» £»£»£»ù´¡Ä£×Óº­¸Ç Qwen2.5-Math (1.5B, 7B) ºÍ Qwen3-4B¡£¡£¡£¡£¡£

1. Ö÷Á÷»ù×¼ÉϵÄÄëѹʽÌáÉý

TTCS ÔÚËùÓгߴçµÄÄ£×ÓÉϾùÈ¡µÃÁËÏÔÖøÓÅÓÚ»ùÏßµÄЧ¹û£¨Ïê¼û Table 1£©£º

Qwen2.5-Math-1.5B£ºTTCS µÄƽ¾ù·ÖµÖ´ï£¬£¬£¬£¬ £¬Ïà±ÈԤѵÁ·Ä£×Ó£¨17.30£©ÌáÉýÁË+24.19 ·Ö£¡×ÝÈ»±ÈÕÕ²âÊÔʱ΢µ÷µÄÇ¿»ùÏß TTRL£¨36.56£©£¬£¬£¬£¬ £¬ÒÀÈ»Óдó·ùÓÅÊÆ¡£¡£¡£¡£¡£Qwen2.5-Math-7B£ºTTCS ƽ¾ù·ÖµÖ´ï£¬£¬£¬£¬ £¬Ïà±ÈÆÕ±éʹÓÃµÄ Self-Consistency£¨32.15£©ÌáÉýÁË+20.39 ·Ö£¬£¬£¬£¬ £¬ÔÙ´Î֤ʵÎú×Ô¶¯µÄ²âÊÔʱѵÁ·Ô¶Ê¤ÓÚ±»¶¯µÄ¶à·²ÉÑù¡£¡£¡£¡£¡£

2. ¹¥¿Ë¸ßÄѶȾºÈüÌâ (AIME)

ÔÚ×îÄÜÌåÏÖÍÆÀíÉÏÏÞµÄ AIME ¾ºÈüÌâÉÏ£¬£¬£¬£¬ £¬TTCS Õ¹ÏÖÁ˼«Ç¿µÄ¹¥¼áÄÜÁ¦£º

AIME 2024£ºÔÚ 1.5B Ä£×ÓÉÏ£¬£¬£¬£¬ £¬TTRL ½öµÃ 13.23 ·Ö£¬£¬£¬£¬ £¬¶ø TTCS Ô¾ÉýÖÁ·Ö£¬£¬£¬£¬ £¬ÌáÉý·ù¶È´ïAIME 2025£ºÔÚ 7B Ä£×ÓÉÏ£¬£¬£¬£¬ £¬TTCS µÖ´ïÁË·Ö£¬£¬£¬£¬ £¬ÏÔÖøÓÅÓÚ TTRL µÄ 14.06 ·Ö¡£¡£¡£¡£¡£ÕâÓÐÁ¦µØÖ¤ÊµÎú£ºµ±²âÊÔÌâÌ«Äѵ¼Ö TTRL µÄα±êǩʧЧʱ£¬£¬£¬£¬ £¬TTCS ͨ¹ýºÏ³ÉÖÐÐÄÄѶȵĿγ̣¬£¬£¬£¬ £¬ÀֳɼܯðÁËͨÍù¸ßÄѶÈÍÆÀíµÄÇÅÁº¡£¡£¡£¡£¡£

3. Ϊʲô TTCS ÓÐÓã¿£¿£¿£¿£¿£¿£¨Éî¶ÈÆÊÎö£©

ΪÁË̽ÌÖÐÔÄÜÌáÉýµÄȪԴ£¬£¬£¬£¬ £¬ÍŶӾÙÐÐÁ˶àά¶ÈµÄÆÊÎö£º

²»µ«½öÊÇÊýѧ£¨·º»¯ÐÔ£©

ʵÑé Q1 ÏÔʾ£¬£¬£¬£¬ £¬ÔÚ AIME ÉϾÙÐвâÊÔʱѵÁ·µÄ TTCS Ä£×Ó£¬£¬£¬£¬ £¬ÔÚMMLU-ProºÍSuperGPQAµÈͨÓÃÁìÓòÍÆÀíʹÃüÉÏҲʵÏÖÁËÐÔÄÜÔ¾Éý£¨Figure 3a£©¡£¡£¡£¡£¡£Õâ˵Ã÷Ä£×Óѧµ½µÄÊÇͨÓõÄÍÆÀíÂß¼­£¬£¬£¬£¬ £¬¶ø·Ç¼òÆÓµÄ¹ýÄâºÏ¡£¡£¡£¡£¡£

¶¯Ì¬ÏÈÉú > ¾²Ì¬Ãûʦ

ÍŶÓʵÑéÓÃÒ»¸ö¸üÇ¿µ«Àο¿µÄQwen2.5-14B-InstructÄ£×Ó×÷Ϊ Synthesizer£¨ÊµÑé Q3£©¡£¡£¡£¡£¡£Ð§¹ûÁîÈ˾ªÑÈ£º¹²½ø»¯µÄ 1.5B Synthesizer (TTCS) ´øÀ´µÄÌáÉý (+5.34) ¾¹È»ÊǾ²Ì¬ 14B Synthesizer (+2.66) µÄÁ½±¶£¡

ÕâÕ¹ÏÖÁËÒ»¸öÉî¿ÌµÄÔ­Àí£ºË³Ó¦Ñ§ÉúÄ¿½ñˮƽµÄÏÈÉú£¬£¬£¬£¬ £¬±È´¿´âˮƽ¸ßµ«²»¶®Òò²ÄÊ©½ÌµÄÏÈÉú¸üÖ÷Òª¡£¡£¡£¡£¡£

Êý¾ÝЧÂʾªÈË

×ÝȻֻʹÓà 10% µÄ²âÊÔÊý¾Ý£¨½ö 3 µÀÌ⣩£¬£¬£¬£¬ £¬TTCS ÔÚ AIME24 ÉϵÄ׼ȷÂʾÍÄִܵï13.33£¬£¬£¬£¬ £¬Ô¶³¬Ò»ÂÉÊý¾ÝÁ¿Ï TTRL µÄÌåÏÖ£¨Figure 4£©¡£¡£¡£¡£¡£ÕâÅú×¢ TTCS Äܹ»¸ßЧµØÕ¥È¡Ã¿Ò»¸ö²âÊÔÑù±¾µÄÐÅÏ¢Á¿¡£¡£¡£¡£¡£

04. ×ܽáÓëÕ¹Íû

TTCS ¿ò¼ÜµÄÌá³ö£¬£¬£¬£¬ £¬ÊǶԡ¸²âÊÔʱÅÌË㡹·¶Ê½µÄÒ»´ÎÖØ¹¹¡£¡£¡£¡£¡£ÍŶÓ֤ʵÎú£ºÔÚ²âÊÔʱ£¬£¬£¬£¬ £¬Ä£×Ó²»Ó¦ÊDZ»¶¯µÄ½âÌâÕߣ¬£¬£¬£¬ £¬¶øÓ¦ÊÇ×Ô¶¯µÄ¿Î³ÌÉè¼ÆÕß¡£¡£¡£¡£¡£

ͨ¹ý Synthesizer ºÍ Solver µÄ¹²½ø»¯£¬£¬£¬£¬ £¬TTCS ÍêÉÆ½â¾öÁ˲âÊÔʱѵÁ·ÖС¸Êý¾Ýϡȱ¡¹ºÍ¡¸ÄѶȶϲ㡹µÄÁ½´óÍ´µã¡£¡£¡£¡£¡£

Õâ²»µ«ÎªÌáÉý´óÄ£×ÓÔÚÖØ´óÊýѧ°Ù´¨Ä¸Ó¤ÓÃÆ·ÓÐÏÞ¹«Ë¾¡¢´úÂëÍÆÀíʹÃüÉϵÄÌåÏÖÌṩÁËÐÂ˼Ð÷£¬£¬£¬£¬ £¬Ò²ÎªÎ´À´×Ô½ø»¯ÖÇÄÜÌ壨Self-Evolving Agents£©¡ª¡ª ¼´Ä£×ÓÄܹ»ÔÚÍêȫδ֪µÄÇéÐÎÖÐͨ¹ý×ÔÎÒ²©ÞÄʵÏÖÒ»Á¬½ø»¯ ¡ª¡ª µÓÚ¨Á˼áʵ»ù´¡¡£¡£¡£¡£¡£