一、技术方案源于哪些需求
目标：给出一个网页 URL，最终得到正文 Markdown，其中不再包含侧边栏/广告/导航。
技术选型：
- readability（readability-lxml）做正文去噪
- markdownify 把清洗后的 HTML 转 Markdown
- 预清理由 BeautifulSoup 负责
- 客户端渲染站点交给 Playwright
Doubao-Seed-Evolving 通过一次对话生成了代码初版；经过提炼的关键实现和踩坑内容，将在下文展开。
二、初版实现
这是我输入提示词后AI给的回答：
以下为核心结构节选：
import requestsfrom bs4 import BeautifulSoupfrom readability import Documentimport markdownify as mdDEFAULT_UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ..."def fetch_url(url, timeout=15, verify_ssl=True):if not url.startswith(("http://", "https://")):url = "https://" + urlheaders = {"User-Agent": DEFAULT_UA}resp = requests.get(url, headers=headers, timeout=timeout, verify=verify_ssl)resp.raise_for_status()return resp.textdef extract_markdown(html, page_url=""):soup = BeautifulSoup(html, "lxml")for tag in soup.find_all(["script", "style", "nav", "aside", "footer"]):tag.decompose()doc = Document(str(soup), url=page_url)content_html = doc.summary()return md(content_html, heading_style="ATX", convert=["table", "pre"])
注意：第一版有个坑——markdownify 不能同传 convert 和 strip，否则运行时直接报错。预清理已处理 script/style，所以删掉 strip 即可。这个修复也是模型在看到报错后给出的。
三、三类网站的实测结果
测试对象涵盖 3 个不同类型的网页，结果在此按实际情况记录：
| 网站类型 | 结果 | 现象 |
|---|
| SSR 技术博客 | ✅ | 完整保留代码块、语言标注和表格 |
| 公众号 | ⚠️ | 标题显示 [no-title]，正文则为 OK |
| 头条（CSR） | ❌ | 正文没有内容，仅出现 # [no-title] |
测试的三个网页中，一个直接成功，一个缺失标题，另一个完全只剩空壳。第一版并没有框架预设中那么糟糕，但实际问题更有意思，因为不同网站分别暴露了不同短板。
四、标题提取退化的关键修复
og:title 不在 readability-lxml 的 Document.summary() 读取范围内，它只提取 </code>。解析 bug 会被微信文章的标题格式触发，结果便是 [no-title]。</p><p></p><p>修复：语义化标题提取应作为一层处理，置于 extract_markdown 之前。</p><p><code><span class="hljs-keyword">def</span> <span class="hljs-title function_">extract_title</span>(<span class="hljs-params">soup, doc</span>):og = soup.find(<span class="hljs-string">"meta"</span>, <span class="hljs-built_in">property</span>=<span class="hljs-string">"og:title"</span>)<span class="hljs-keyword">if</span> og <span class="hljs-keyword">and</span> og.get(<span class="hljs-string">"content"</span>):<span class="hljs-keyword">return</span> og[<span class="hljs-string">"content"</span>].strip()tw = soup.find(<span class="hljs-string">"meta"</span>, attrs={<span class="hljs-string">"name"</span>: <span class="hljs-string">"twitter:title"</span>})<span class="hljs-keyword">if</span> tw <span class="hljs-keyword">and</span> tw.get(<span class="hljs-string">"content"</span>):<span class="hljs-keyword">return</span> tw[<span class="hljs-string">"content"</span>].strip()h1 = soup.find(<span class="hljs-string">"h1"</span>)<span class="hljs-keyword">if</span> h1 <span class="hljs-keyword">and</span> h1.get_text(strip=<span class="hljs-literal">True</span>):<span class="hljs-keyword">return</span> h1.get_text(strip=<span class="hljs-literal">True</span>)<span class="hljs-keyword">return</span> doc.short_title()</code></pre><h2>五、关键修复二：客户端渲染站点的空壳</h2><p>头条这类站点正文由 JS 异步加载，requests 拿到的只是 <code><div id="root"></code> 空壳。</p><p></p><p>修复：通过 --render 参数启用 Playwright 无头浏览器，完成渲染等待后再开始提取。</p><p><code><span class="hljs-keyword">from</span> playwright.sync_api <span class="hljs-keyword">import</span> sync_playwright<span class="hljs-keyword">def</span> <span class="hljs-title function_">fetch_rendered</span>(<span class="hljs-params">url</span>):<span class="hljs-keyword">with</span> sync_playwright() <span class="hljs-keyword">as</span> p:browser = p.chromium.launch(headless=<span class="hljs-literal">True</span>)page = browser.new_page()page.goto(url, wait_until=<span class="hljs-string">"networkidle"</span>)html = page.content()browser.close()<span class="hljs-keyword">return</span> html</code></pre><p>另：微信/头条图片是懒加载，真实地址在 data-src，需补一段把 data-src 回填到 src，否则全文裂图。</p><h2>六、最终使用方法</h2><p><code>python extract_article.py https://juejin.cn/post/xxx -o article.mdpython extract_article.py https://www.toutiao.com/xxx --render -o toutiao.md</code></pre><p>静态站不需要额外依赖，动态站加 --render。</p>
</div>
<div class="tips_content">郑重声明：本站发布内容宗旨在传播更多信息，仅提供查阅，与本站立场无关，不拥有所有权，不承担相关法律责任。不具有任何效益，仅供参考。如果需要专业知识建议，请咨询相关专业人士。如有侵权请联系邮箱。一经查实，立即删除！</div>
<!-- -->
<ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-5776344680640037" data-ad-slot="4494831787" data-ad-format="auto" data-full-width-responsive="true"></ins>
<script> (adsbygoogle = window.adsbygoogle || []).push({});</script>
<div class="content_btn_item clearfix">
<span class="fl">喜欢(0)</span>
</div>
<!-- -->
<div class="swiper_page clearfix">
<div class="fl_item fl">
<a href="/news/3432295.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a696f4f705431.jpg">
</span>
<span class="text">上一篇</span>
<p class="text_overflow_2">盗墓笔记启程积尸地怎么过 积尸地副本打法流程攻略</p>
</a>
</div>
<div class="fr_item fr">
<a href="/news/3432297.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a696f7b8a56f1.webp">
</span>
<span class="text">下一篇</span>
<p class="text_overflow_2">重生之我成为模型 第三篇 · 我只提供可能性，答案由你们翻译</p>
</a>
</div>
</div>
</div>
<ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-5776344680640037" data-ad-slot="4494831787" data-ad-format="auto" data-full-width-responsive="true"></ins>
<script> (adsbygoogle = window.adsbygoogle || []).push({});</script>
<!-- 猜你喜欢 -->
<div class="like_box lm_box">
<p>猜你喜欢</p>
<ul class="clearfix">
<li>
<a href="/news/3432759.html">
<div>
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69905ba3b391.webp">
</div>
<p class="text_overflow_2">盗墓笔记启程角色培养推荐 盗墓笔记启程职业选择详情一览</p>
</a>
</li>
<li>
<a href="/news/3432758.html">
<div>
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a699054ce2761.webp">
</div>
<p class="text_overflow_2">盗墓笔记启程神殿副本攻略分享 盗墓笔记启程神殿如何通关</p>
</a>
</li>
<li>
<a href="/news/3432757.html">
<div>
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69904e382781.webp">
</div>
<p class="text_overflow_2">盗墓笔记启程古城攻略盘点 盗墓笔记启程古城通关详情指南</p>
</a>
</li>
<li>
<a href="/news/3432756.html">
<div>
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a699046063361.webp">
</div>
<p class="text_overflow_2">盗墓笔记启程冒险玩法解析 盗墓笔记启程冒险玩法详情推荐</p>
</a>
</li>
<li>
<a href="/news/3432755.html">
<div>
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a699020520f11.webp">
</div>
<p class="text_overflow_2">《怪物猎人：荒野》攻略——大锤武器特性分享</p>
</a>
</li>
<li>
<a href="/news/3432754.html">
<div>
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a699017aff331.webp">
</div>
<p class="text_overflow_2">遗忘之海SSR角色抽哪个好 遗忘之海SSR角色抽取教程</p>
</a>
</li>
</ul>
</div>
</div>
<div class="r_item">
<!-- 大家都在看 -->
<div class="lm_box">
<p>大家都在看</p>
<dl class="img_item">
<dd>
<a class="clearfix" href="/news/3429903.html">
<div class="img_icon">
<img lazy-src="https://www.0405.net/uploads/20260728/logo_6a6817f10dd1d1.webp">
</div>
<div class="content">
<p class="title text_overflow_2">GitHub周趋势2026W25 | Headroom 压缩 95% Token、NVIDIA 开源 AI Agent 安全扫描器、…</p>
<p class="time">2026-07-28</p>
</div>
</a>
</dd>
<dd>
<a class="clearfix" href="/news/3422497.html">
<div class="img_icon">
<img lazy-src="https://www.0405.net/uploads/20260725/logo_6a63fc43247d61.webp">
</div>
<div class="content">
<p class="title text_overflow_2">腾讯云ADP 4.0海外版发布：要把企业级智能体带到全球市场 | 最前线</p>
<p class="time">2026-07-25</p>
</div>
</a>
</dd>
<dd>
<a class="clearfix" href="/news/3423548.html">
<div class="img_icon">
<img lazy-src="https://www.0405.net/uploads/20260725/logo_6a6447b970be41.webp">
</div>
<div class="content">
<p class="title text_overflow_2">综艺《歪果仁研究协会》的剧情介绍</p>
<p class="time">2026-07-25</p>
</div>
</a>
</dd>
<dd>
<a class="clearfix" href="/news/3425870.html">
<div class="img_icon">
<img lazy-src="https://www.0405.net/uploads/20260726/logo_6a659da7d83c01.webp">
</div>
<div class="content">
<p class="title text_overflow_2">Sekimio好玩吗 Sekimio玩法简介</p>
<p class="time">2026-07-26</p>
</div>
</a>
</dd>
<dd>
<a class="clearfix" href="/news/3429201.html">
<div class="img_icon">
<img lazy-src="https://www.0405.net/uploads/20260728/logo_6a67e798560d21.webp">
</div>
<div class="content">
<p class="title text_overflow_2">动漫 AI 提示词评审会</p>
<p class="time">2026-07-28</p>
</div>
</a>
</dd>
</dl>
</div>
<ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-5776344680640037" data-ad-slot="4494831787" data-ad-format="auto" data-full-width-responsive="true"></ins>
<script> (adsbygoogle = window.adsbygoogle || []).push({});</script>
</div>
</div>
</div>
</div>
<!-- 底部 -->
<div class="footer_template">
<div class="w12_box">
<div class="l_item">
<a href=""><img src="/newindex/images/f_logo.png" alt=""></a>
</div>
</div>
<div class="link_item">
<div class="w12_box">
<em>友情链接:</em>
<a href="https://www.kofbobo.net/" title="波波三国" target='_blank'>波波三国</a>
<a href="http://www.diablofans.com.cn/" title="暗黑破坏神中文网" target='_blank'>暗黑破坏神中文网</a>
<a href="https://yeodown.jiyx.com/" title="易欧交易所" target='_blank'>易欧交易所</a>
<a href="https://www.alixixi.com/" title="阿里西西" target='_blank'>阿里西西</a>
</div>
</div>
<div class="w12_box">
<p>Copyright(C) 2020-2026 jiyx.com All Rights Reserved 联系方式：ds0405@163.com</p>
<p><a href="https://beian.miit.gov.cn/">湘ICP备2022015115号-2</a></p>
</div>
</div></body>
<script src="/index/js/jquery.js" type="text/javascript" charset="utf-8"></script>
<script src="/index/js/lazy.js" type="text/javascript" charset="utf-8"></script>
<script src="/index/js/tips_scroll.js" type="text/javascript" charset="utf-8"></script>
<script src="/redirect/skip.js" type="text/javascript" charset="utf-8"></script>
<script type="text/javascript">
var _paq = window._paq = window._paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//tongji.zhangwan.net/";
_paq.push(['setTrackerUrl', u+'matomo.php']);
_paq.push(['setSiteId', '9']);
// Add this code below within the Matomo JavaScript tracker code
// Important: the tracker url includes the /matomo.php
var secondaryTrackerUrl = u+'matomo.php';
var secondaryWebsiteId = 27;
// Also send all of the tracking data to this other Matomo server, in website ID 77
_paq.push(['addTracker', secondaryTrackerUrl, secondaryWebsiteId]);
// That's it!
var d=document,
g=d.createElement('script'),
s=d.getElementsByTagName('script')[0];
g.type='text/javascript';
g.async=true;
g.src=u+'matomo.js';
s.parentNode.insertBefore(g,s);
})();
</script>
</html>
