在上一部分，我们学习了爬虫的基础知识。现在，我们使用 Java + Jsoup 编写爬虫，抓取天气、新闻和商品信息。

一、爬取天气信息

使用 Jsoup 获取天气

我们可以使用 Jsoup 解析天气网站的 HTML，获取当前天气信息。

示例代码
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class WeatherScraper {
    public static void main(String[] args) {
        String city = "beijing";
        String url = "https://wttr.in/" + city + "?format=%C+%t";
        try {
            Document doc = Jsoup.connect(url).get();
            System.out.println(city + " 当前天气：" + doc.text());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}
执行结果

image-VKAbtxQiDZksoHUCVDBpTJzxMwAlREZd.png

beijing 当前天气：Haze +13°C

二、 爬取新闻标题

使用 Jsoup 获取新闻 爬取新浪新闻首页的头条新闻标题。

image-LepaUtzzvvSAdXBCkhWRRYPYhMSkwLID.png

示例代码
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;

public class NewsScraper {
    public static void main(String[] args) {
        String url = "https://news.sina.com.cn/";
        try {
            Document doc = Jsoup.connect(url).get();
            Elements headlines = doc.select("h3");  // 选择新闻标题
            for (int i = 0; i < Math.min(5, headlines.size()); i++) {
                System.out.println((i + 1) + ". " + headlines.get(i).text());
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}
执行结果

image-oKTRshCQEQRHyKyHYSFNedciSxKHPNCO.png

1. 蜂鸟
2. 前主编
3. 新浪新闻在场
4. 极空间私有云 Z2pro

3. 爬取商品价格

使用 Selenium 爬取淘宝商品价格 由于淘宝页面是动态加载的，我们需要使用 Selenium 进行爬取。

安装 Selenium

  1. 下载 ChromeDriver：确保本地安装了 Chrome 浏览器，并下载 ChromeDriver（点击下载）。

  2. 引入 Selenium 依赖（如果使用 Maven）：

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.8.0</version>
</dependency>
示例代码
import io.github.bonigarcia.wdm.WebDriverManager;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

import java.util.List;

public class TaobaoScraper {
    public static void main(String[] args) throws InterruptedException {
        // WebDriverManager 自动下载并管理 ChromeDriver
        WebDriverManager.chromedriver().setup();
        WebDriver driver = new ChromeDriver();
        driver.get("https://s.taobao.com/search?page=1&q=iPhone16pro&tab=all");
        // 等待页面加载（可以用显式等待替代）
        Thread.sleep(3000);
        // 获取所有商品的 DOM 元素
        List<WebElement> items = driver.findElements(By.cssSelector(".doubleCard--gO3Bz6bu"));

        for (WebElement item : items) {
            try {
                // 商品标题
                String title = item.findElement(By.cssSelector(".title--qJ7Xg_90 span")).getText();

                // 商品价格
                String price = item.findElement(By.cssSelector(".innerPriceWrapper--aAJhHXD4")).getText();

                // 付款人数
                String sales = item.findElement(By.cssSelector(".realSales--XZJiepmt")).getText();

                // 商品链接
                String link = item.findElement(By.tagName("a")).getAttribute("href");

                // 打印商品信息
                System.out.println("标题: " + title);
                System.out.println("价格: " + price);
                System.out.println("销量: " + sales);
                System.out.println("链接: " + link);
                System.out.println("--------------------------");
            } catch (Exception e) {
                //System.out.println("获取商品信息失败: " + e.getMessage());
            }
        }
        // 关闭浏览器
        driver.quit();
    }
}
执行结果

image-ugFUycOymbnljhCevMJrYqVwcRTPPsbv.png

标题: 现货速发Apple苹果 iPhone 16 Pro 手机 支持全网通5G 全新 美版有锁
价格: 5264
.00
销量: 100+人付款
链接: http://click.simba.taobao.com/cc_im?p=iPhone16pro&s=1746403449&k=1793&e=WOfj7Viswq85G4F6TbNcv0Q9wKP5gYyY9v5X8Y55dZh3UU%2B0yDM0l68vVMcPU3X4xt%2BMDnNk7FHkBB%2FJdtNBpEATPE68fUr5Z90PYJ3ekDgg97eLXlsQAF1jwHzTuG%2BK65RpO%2BBWdIIv2jvkFkOPWVSfJCEBXXCiMFV9JEQtQPsSRXUHyomLuCp6ATlEODgaf8q2%2BHPkKQOBhUpf%2Fnr70UPRNrStQfhH%2BUanydqcqepcdqPOgGKRlDRpPw4a7SIhkEOmf7UWnSVlYHMriLICE0eP9QNQojJGgr2F9LK2RNdmpFG74QMWQg5BHMqhMthqpsi%2BPskaw6SQjtHhXQ4fA8T12tDSMEg%2FK3sOJXv0q3eE5ENIwu7g1ySSaHKNsjmWI%2FcpdYGcWfNQbQXuvo5Cogn5cLp80cHixtCtUtL2yTECUh1UaKwpnYZ%2FnPkgT0C3Jpw1ehM6MxZPg%2BnmyO5dlLX3%2BGslmgr%2B%2FcTWd7vwj4J9%2FO1P8%2FjrsTzzvBXDCJQgMQGlhFxTjTnNOYZSQbdDjgQ8Tu9%2FbFOUzxRDGX9DtO5%2Fz0917Tkjy148D3FfgcY0glyGzRpqBIfOQML%2Fj61M2MVjppPlra2uoEqdVHgeUAwhSZq%2B5EMl1c0c52hARdUo6LtssAwuSXlNaODlr%2FG6%2F7I8Km25vkVpr2Tu2QMuXc6ND6oCpWKk7kyrhT4R0IZGV%2Bx0wt%2BDowWOFDbdQzpVvv7tnWt9dWg2TbD4K9T8gwCRNHivqHQ84Koa2aFf%2BjbnS%2BMhS%2BnsmdblJ0ZE4YX%2Bh%2Fm7kBaA67Umv9%2B74OPCk5PfFRM5Aow72J1NPgdAzG8JAjtyAqqb98SWoWPwKgwHyg6VlOEnJpHIakr%2BQoHSqytYTl6ana9Bw2kBwEP%2Bj5OgMP6jxqElxe7lz5eAnRArKOR7inTzTwKsmEWgyfreIzUfCd9Qu%2BhlDcdS9qxLAxXt%2FUlek7Qzz6oRVmIDeBmPTKTT%2BsIRhu8Uv5tSUI9%2FyJz2wwnNbYJWe6HgwbBlvl94DLJvUkHA3UgZxLqnU3kqkKuaA8p9VIQW46smk2jQOa9vwrOdlWxe4zipLsGHK45IwNNpSggc8SaCNV1Hsq6kYDqo%2FppVsljSvR5p29dXpdvfO%2FpUFOR0x1A%2BP2Qvtthb6DM39t%2BNT9gk%2FJlCAIaKB3387U%2Fz%2BOuxvqvwzVJ88gnW84ZCGhAIh6tfMLkyX%2F1dNcxjJTHR4bSA2BFqjwM72s05hlJBt0OOLdIRA5e0KHLu1rRG8zut4B9RdpuimYsIE%2FUYvjHlWkD3fI%2BGIGeEHCAgoxKwq1U7qOj3TGHitHWivxV%2BTgfH2%2Be6nd11Bf88sg%2F1esCTVeHR%2FlSI1ClcLqK%2FFX5OB8fbfTlusX9Lv%2B1%2FsP3FMmRSa%2FuOPT%2FoZHzq50oNCnCDpcMOZzR6kMxiMoAGnJUWCjYq9TE1Ei1iqxtdGE1mbsyvCgoKCkjhdFA7RMw22GotXE78Jgf9cavSjOkakGfTcDflB79M%2FqAPGA2Vf4i27zrnEbj5pG5i9IaCNcxjJTHR4bTwqZwDscb0lAumrMn0M16kMZlCjreuDLfqJTXENkR4BJVDCxXTNS5SWMsPH85KqCgBmScmEYK6DDc3d%2F7ySuAZhGBl2%2BIVka4pCKjnRcUb0kdui1vj%2BG1Aan7YWJkYZ5LBzM7Y8E7JdtZlPwIfjqAWSc%2FwQ%2BTFBcAFPJxS%2FnQ4IgCLcRO7OewZ7ta0RvM7reBemRXL%2FP2UKmRiBndrOunXDlR12XdAizJieQptECWOwStXLcCHDWlb&mm_sceneid=3_0_2420640192_0&spm=a21n57.1.item.1
--------------------------
标题: 12期免息Apple/苹果 iPhone 16 Pro 支持移动联通电信5G 双卡双待
价格: 7264
.00
销量: 1000+人付款
链接: http://click.simba.taobao.com/cc_im?p=iPhone16pro&s=1746403449&k=1665&e=OXq9Bf3E88E5G4F6TbNcv0Q9wKP5gYyY0WT66UTytRdI3fBsMos5YT2OKX9wFcEWxt%2BMDnNk7FEcvsJIFwPxCiR2ALzUqqB6B9UYXiwzzd31Z2ccUmcUTWFg31ySNKJk65RpO%2BBWdIIv2jvkFkOPWVSfJCEBXXCiMFV9JEQtQPsSRXUHyomLuCp6ATlEODgaf8q2%2BHPkKQOBhUpf%2Fnr70UPRNrStQfhH%2BUanydqcqepcdqPOgGKRlDRpPw4a7SIhkEOmf7UWnSVlYHMriLICE0eP9QNQojJGgr2F9LK2RNdmpFG74QMWQg5BHMqhMthqpsi%2BPskaw6SQjtHhXQ4fA8T12tDSMEg%2FK3sOJXv0q3eE5ENIwu7g1ySSaHKNsjmWI%2FcpdYGcWfO7FlOPP15R%2FlMtl5qk8%2BoRi5OcJSx11%2BDoKP7OnjokjbxyfnZHCJ%2FTNQF7nYrDnxuRJyU3SvQD6fW4%2F8m4UiQn%2FExUikbah149qbUaUHOqq5a3AFHO6Hmdwr9gy10iXqaOacXYIGReSuRo0O46nl6chG%2F56EwKry1AultgsSD2TlIAK4kKe2WE7%2B%2BEQ0PvF%2FsCatL9N8IbOcsmoKpqp8KrWAET1SFi407BQPiA7vrXNgt1P6kJZg3jYKf4Ll0a1u25f6Pt%2FkQOsUvEIm5XwPyMfBWgGCIDSfMSlLDNG8s%2BGa5ikb1PVNn1PyVF2YetbCHR8kUs5IlCJoFmH0sWI%2BCx74Feg5LesrmeoKI1mTHAh%2B%2BkQXk%2BAqlS%2BWVEAQuL9eSTKISvPn78QR%2Bjj2HyUCz86GZzVye0B38my6qU0H%2Fo2VgYy4iMMuJ8piNuBpdK7wvTywX4SMvYVdoaDk6tnlhhDyg1wF%2B2rahOKP0%2FXS2rEPChOSK%2F%2FpTWp0jHtbvfFWLRtEi%2Fbp3kkkBs8BN6TSaOSMG9vX0F9eKZfn0QJNdSEdH54T7iunelM3HcrVSL9bro%2BdWYYcg4c0mUj57qe0c9iBAgbcj%2B0KVEsDU4m7pTs5xs6elQwDHXfdJlrpmF2pMBCwm6%2FNMR1grpGTSTpHtzPZheQ6l%2FmVjOy9SgMvD7P2MSj83kIkDJ4jU5geAxbio9GGCOiBUEZO9j9YDGs1Y%2BPam1GlBzqqu8DSxNGZ0htpFEi0Gtb4N05UyKMH4HOLhl%2Fy%2F5z0s0boUDGix%2F35e%2B3GwqWv9oy6scixw4ZmHGmiud5qWFlljjQPP01FYdiOsxJ4F%2FTNg35Lq8EnduBZAn8gTA6I1TizhQADSIfoqi%2BUc9smrkcE3xruo84gZzwQOhXBKbPBERTV4LuzoIWsXbRz2yauRwTfGu6jziBnPBA4BE%2FEb%2BL8cvQg45F99fQVfl4JfkvWnGmFbL6RczIURCI9DhALNaw3wq4UvBRP%2BscYDlLK1iVytW5QXre99Mel7z3KWMSthd7xxljwbVt7b1Ctnz9OZloxUJsF%2BzPQDhHHUsM9BhBwHTfNDoZI%2Frs7uKzwhh6%2FguZaXHp0pafIF3j03qzSCDLpAp6Ul3P6H1l7EO29ErC6coPyVF2YetbCHR8kUs5IlCJoFmH0sWI%2BCx74Feg5LesrmXuk3w9RyRZcz0iL2COpEetJFMMKu%2B%2BnZBXRkmfSv8%2Bo7I2fuI5Pd6ZYAf39bs6fFxtH6YUPhaG7aPcgemoBfF&mm_sceneid=7_0_6803483680_0&spm=a21n57.1.item.4
--------------------------
标题: 顺丰正常发货Apple/苹果 iPhone 16 Pro/16 Pro Max 24期免息
价格: 7564
.00
销量: 1000+人付款
链接: http://click.simba.taobao.com/cc_im?p=iPhone16pro&s=1746403449&k=1945&e=bGzv5DNSeEI5G4F6TbNcv0Q9wKP5gYyY0WT66UTytRdI3fBsMos5YT2OKX9wFcEWxt%2BMDnNk7FHuZ5z3Zt6k1PsXX3T8y2HfldRHoBuThYf1Z2ccUmcUTWFg31ySNKJk65RpO%2BBWdIIv2jvkFkOPWVSfJCEBXXCiMFV9JEQtQPsSRXUHyomLuCp6ATlEODgaf8q2%2BHPkKQOBhUpf%2Fnr70UPRNrStQfhH%2BUanydqcqepcdqPOgGKRlDRpPw4a7SIhkEOmf7UWnSVlYHMriLICE0eP9QNQojJGgr2F9LK2RNdmpFG74QMWQg5BHMqhMthqpsi%2BPskaw6SQjtHhXQ4fA8T12tDSMEg%2FK3sOJXv0q3eE5ENIwu7g1ySSaHKNsjmWI%2FcpdYGcWfO7FlOPP15R%2Fo3vmUGpHDfMD9b2VOYKtXVtB2iNwE2L4UlB5B%2FBrZ0%2F5QOjDW2gt435AQSiFD%2FD7uxucKhEVhua%2F%2FU9kbV59pC%2BJ2nxy4k%2Bsh%2FM62vYHFP5pOozH4tMEmBgX4QzkGqmK87ZXO58DElE8cYcZoRthkoMz4im0%2BhFasMrqiIU%2BMQ6n6Glgu0d7cMCO5%2F0c27Cb28l3lEO237CLwWJNZdITpMavkh4Oh7KapHtLzjK0x3ADmtZUlBbkggrA%2BPakesC9GlMrzK4Zpf%2Bj9%2F8Ts1wPF5GKYxT0VxaR%2FTYSL%2BF%2FRxr9VNd4JtG31tIDCs87etOZncdQRg%2FuO5y0U0cG%2BWJo6vC0Kux%2BCxGr9tRYtiUcKU7B8nNqusd7JIg%2FCzUeEbOL1vubwF%2FaVPG2iUed%2BcXWSgYAZTXyvrTQe0Bs7RNMhbjiPnp3fd0nupXbKhqXZodoY%2Ff%2FE7NcDxe%2Bs%2BxlW%2Bn4e%2FGsjAykK3uTTDPT5wOkqb87GJPTxfsxzvOxeYXIjGm0EhW%2Bi3kKLT%2BLt4%2Bf7Wo3tr%2F2GbrTAEU%2FKVa8VLtH0wFmdxPIW8TIFgJjA2JttNcCAEUiI2G72FaN79chFfzH2ar3EihK96rgnj6UhB3cTkNCggEv2DY0B%2BodJ7whEHLU8NLlCMaNfZiNXEcUXWZRTHGB%2B5GclHOjjn1yZAsMUotwg8eDdjav9qMkYmFcIK%2Frr0FuUxltlLy1%2FDyVEYal3vMvRuxyICaGbOHyxx8BOpZ6W3hluYfZoIsnohSFSvWE4sPoESTrYZntt%2FMxAO3BwOZjJPMRSYgqEmSOzL6uoQ9qgTitBPwDzddIQ%2BIo5PQNvI9NUTvtJftAU4qwfcp1bvdUSPLYKynuN3EBMPnyM%2BfGydbyPqws9mrYKBnkZTzvosT4wVFdKpW8tz%2Fw8k76kChjsuImS01r7zmZyplIRNmMHBLXmoRyrDrZO1AtipusF3rzSup1O1ynH5RFkK1wbz7rtNdenyl1aIwtWYFv%2FPKOMQcUNGk8fVb7m8Bf2lTxtKN1wAh%2FZty5%2BaDTMByamFvOH5MlvtKalvubwF%2FaVPG1qjGrl7G1I0IbUAdpW57V%2FqmdrPD8XPN13ifesChmVJAFd9GpR4iSP5%2BDKgrr4gYpt%2BQTlSCwub8FRcREZwZVdK4EVzRccdnSQSZeErcqqY7wRQhY17Rw%2FwmB%2F1xq9KM6RqQZ9NwN%2BUHv0z%2BoA8YDZV%2FiLbvOucRuPmkbmL0hoI1zGMlMdHhtPCpnAOxxvSUC6asyfQzXqSX4CNNM468Kq%2FwbhtQnUdGnuF%2FVWTLEmMUZsGNWCw69FdV86e4x7iSQgB9dK%2Fgfke%2FIbOMaG2EWXBBEBViLg9qhGBl2%2BIVka4pCKjnRcUb0kxx2hfuU9%2FHVMHbUNfrPj%2B8LygqwtBzzhKI%2BUFdeLgljA4LShdNfhaHBn8ha15%2FG50IhdaiLtEGKQeixIdkY8Ghy%2F2awvgpXXd%2BD8rThGB2XHrCo8qzpZqFFOQd8JCvOS3i22sXfijzEcDQuFRWg4R95qN%2FQpTUrw%3D%3D&mm_sceneid=1_0_96812084_0&spm=a21n57.1.item.6
--------------------------